Managed data ingestion

Data Hub ingestion with customer-owned connectors, Meltano/Singer integration, and backend-owned execution.

Managed ingestion moves records between source and destination connections. Users configure and operate it through Data Hub in the Portal. The BSQAI API owns configuration, authorization and run history; Temporal coordinates execution and schedules; Kubernetes runs a trusted coordinator and separate customer-owned connector containers.

Connector definitions describe immutable image versions, capabilities and JSON Schemas. A connection combines one definition version with configuration and separately encrypted credentials. An ingestion selects source and destination connections, streams and replication settings. Runs record execution outcomes and sanitized diagnostics.

Why It Is Used

BullSequana AI provides one tenant-aware control plane for connector registration, data movement and operations. Teams can add connector implementations without a platform release or a source-specific Portal form. Customers own their connector images, dependencies and destination-write behavior; the platform owns the execution contract and lifecycle.

Learn More

Meltano and Singer integration

MeltanoHub/Singer taps and targets can run inside connector images adapted to the platform's HTTPS execution contract. The supplied PostgreSQL and S3 examples demonstrate that packaging; they are not a built-in connector catalogue or a promise that every upstream connector is supported.

The coordinator does not install plugins or execute the Meltano CLI. Other implementations can use the same contract. The deployment component and several resource names retain the implementation identifier meltano; there is no separate Meltano web UI to sign into.

Execution and data flow

The control plane starts and monitors execution without carrying record batches:

Records and committed checkpoints follow a separate path:

Temporal carries execution references, not record batches or plaintext credentials. Connector Pods receive only their side's runtime configuration and credentials, with no Kubernetes service-account token or checkpoint volume. The coordinator commits progress after source completion and destination success. Partial writes can still be replayed; exactly-once delivery depends on the connector and destination, not just the workflow engine.

Managed ingestion is separate from user-authored Kubeflow Pipelines and from the lakehouse Data Catalog. It does not import pipeline DAGs or register ingested datasets as governed catalog tables automatically.

Deployment and operation

The proai/meltano component provisions execution resources in data-ingestion by default. The connector worker and private runtime API run alongside the BSQAI API in the backend namespace. They reuse the platform's Temporal service. The execution namespace is configurable through the component configuration.

Before handing Data Hub to users, operators provide:

a registry from which Kubernetes can pull the registered images, including any required pull credentials;
a writable shared checkpoint volume, using the site's supported RWX storage class and workload security settings;
the checkpoint backup bucket, credentials and trusted object-store CA;
approved source/destination endpoint access, including connection tests and discovery as well as ingestion runs; and
tenant-bound workload principals for unattended schedules when personal execution identities are unsuitable.

Customer connector egress defaults to DNS only. Endpoint access needs explicit operator-managed network grants and matching native database or storage permissions. Standard Kubernetes NetworkPolicy does not implement hostname-based rules. A dedicated execution namespace alone does not prove tenant isolation; verify allowed and denied paths with the deployed network implementation.

Runtime connection credentials and registry pull credentials serve different purposes. Storing a database password in Data Hub does not authorize image pulls. Digest pinning identifies an image; it does not establish that customer code is trustworthy.

Checkpoint backup and recovery

The meltano-state PVC holds progress separated by tenant and ingestion. The meltano-state-backup CronJob backs up checkpoints to the configured S3-compatible bucket, meltano-state by default, every five minutes by default. MELTANO_S3_ACCESS_KEY and MELTANO_S3_SECRET_KEY configure that backup identity, not users' source or destination connections.

Inspect the deployed resources, adjusting the namespace for the installation:

kubectl get jobs,pods,pvc,cronjobs -n data-ingestion
kubectl describe cronjob meltano-state-backup -n data-ingestion

Check backup readability as well as Job completion. A checkpoint backup is not a backup of destination records, connector images, encrypted connection configuration or the encryption keys needed to read that configuration. Protect those systems separately. See backup and recovery and Data Hub recovery.

Interacts With

Portal and BSQAI API, for configuration, operations and diagnostics.
Keycloak and OpenFGA, for tenant membership, functional permissions and resource grants.
Temporal, for durable execution and scheduling.
Kubernetes, for coordinator Jobs, connector Pods and network controls.
PostgreSQL, for application metadata, encrypted credentials and run history.
Rook-Ceph or external S3-compatible storage, for checkpoint backups and customer-configured data paths.