# Managed data ingestion (/docs/data/components/managed-ingestion)



## Component Category [#component-category]

Data ingestion and integration

## Component Description [#component-description]

Managed ingestion moves records between source and destination connections. Users configure and operate it through [Data Hub](/docs/use/portal/data-hub) in the Portal. The BSQAI API owns configuration, authorization and run history; Temporal coordinates execution and schedules; Kubernetes runs a trusted coordinator and separate customer-owned connector containers.

Connector definitions describe immutable image versions, capabilities and JSON Schemas. A connection combines one definition version with configuration and separately encrypted credentials. An ingestion selects source and destination connections, streams and replication settings. Runs record execution outcomes and sanitized diagnostics.

## Why It Is Used [#why-it-is-used]

BullSequana AI provides one tenant-aware control plane for connector registration, data movement and operations. Teams can add connector implementations without a platform release or a source-specific Portal form. Customers own their connector images, dependencies and destination-write behavior; the platform owns the execution contract and lifecycle.

## Learn More [#learn-more]

* [Use Data Hub](/docs/use/portal/data-hub)
* [Build and register a connector](/docs/develop/data-connectors)
* [BSQAI API reference](/docs/reference)

## Meltano and Singer integration [#meltano-and-singer-integration]

MeltanoHub/Singer taps and targets can run inside connector images adapted to the platform's HTTPS execution contract. The supplied PostgreSQL and S3 examples demonstrate that packaging; they are not a built-in connector catalogue or a promise that every upstream connector is supported.

The coordinator does not install plugins or execute the Meltano CLI. Other implementations can use the same contract. The deployment component and several resource names retain the implementation identifier `meltano`; there is no separate Meltano web UI to sign into.

## Execution and data flow [#execution-and-data-flow]

The control plane starts and monitors execution without carrying record batches:

<Mermaid
  chart="flowchart TD
    USER[&#x22;Portal Data Hub or API client&#x22;] --> API[&#x22;BSQAI API: authorization and configuration&#x22;]
    API --> DB[&#x22;PostgreSQL: metadata, encrypted credentials and run history&#x22;]
    API --> TEMP[&#x22;Temporal workflows and schedules&#x22;]
    TEMP --> WORKER[&#x22;Backend connector worker&#x22;]
    WORKER -->|launch and monitor| JOB[&#x22;Trusted coordinator Job&#x22;]
    JOB -->|HTTPS configuration request| RUNTIME[&#x22;Private backend runtime API&#x22;]
    RUNTIME --> DB
    JOB -. starts and configures .-> PODS[&#x22;Separate source and destination connector Pods&#x22;]"
/>

Records and committed checkpoints follow a separate path:

<Mermaid
  chart="flowchart TD
    EXTERNAL[&#x22;Source system&#x22;] --> SOURCE[&#x22;Source connector Pod&#x22;]
    SOURCE -->|records over HTTPS| JOB[&#x22;Trusted coordinator Job&#x22;]
    JOB -->|records over HTTPS| TARGET[&#x22;Destination connector Pod&#x22;]
    TARGET --> DEST[&#x22;Destination system&#x22;]
    JOB -->|commit after successful transfer| STATE[&#x22;Tenant and ingestion checkpoint on shared PVC&#x22;]
    STATE --> BACKUP[&#x22;Checkpoint backup CronJob&#x22;]
    BACKUP --> S3[&#x22;S3-compatible backup bucket&#x22;]"
/>

Temporal carries execution references, not record batches or plaintext credentials. Connector Pods receive only their side's runtime configuration and credentials, with no Kubernetes service-account token or checkpoint volume. The coordinator commits progress after source completion and destination success. Partial writes can still be replayed; exactly-once delivery depends on the connector and destination, not just the workflow engine.

Managed ingestion is separate from user-authored Kubeflow Pipelines and from the lakehouse Data Catalog. It does not import pipeline DAGs or register ingested datasets as governed catalog tables automatically.

## Deployment and operation [#deployment-and-operation]

The `proai/meltano` component provisions execution resources in `data-ingestion` by default. The connector worker and private runtime API run alongside the BSQAI API in the backend namespace. They reuse the platform's Temporal service. The execution namespace is configurable through the component configuration.

Before handing Data Hub to users, operators provide:

* a registry from which Kubernetes can pull the registered images, including any required pull credentials;
* a writable shared checkpoint volume, using the site's supported RWX storage class and workload security settings;
* the checkpoint backup bucket, credentials and trusted object-store CA;
* approved source/destination endpoint access, including connection tests and discovery as well as ingestion runs; and
* tenant-bound workload principals for unattended schedules when personal execution identities are unsuitable.

Customer connector egress defaults to DNS only. Endpoint access needs explicit operator-managed network grants and matching native database or storage permissions. Standard Kubernetes NetworkPolicy does not implement hostname-based rules. A dedicated execution namespace alone does not prove tenant isolation; verify allowed and denied paths with the deployed network implementation.

Runtime connection credentials and registry pull credentials serve different purposes. Storing a database password in Data Hub does not authorize image pulls. Digest pinning identifies an image; it does not establish that customer code is trustworthy.

### Checkpoint backup and recovery [#checkpoint-backup-and-recovery]

The `meltano-state` PVC holds progress separated by tenant and ingestion. The `meltano-state-backup` CronJob backs up checkpoints to the configured S3-compatible bucket, `meltano-state` by default, every five minutes by default. `MELTANO_S3_ACCESS_KEY` and `MELTANO_S3_SECRET_KEY` configure that backup identity, not users' source or destination connections.

Inspect the deployed resources, adjusting the namespace for the installation:

```bash
kubectl get jobs,pods,pvc,cronjobs -n data-ingestion
kubectl describe cronjob meltano-state-backup -n data-ingestion
```

Check backup readability as well as Job completion. A checkpoint backup is not a backup of destination records, connector images, encrypted connection configuration or the encryption keys needed to read that configuration. Protect those systems separately. See [backup and recovery](/docs/deployment/playbooks/backup-and-disaster-recovery#managed-ingestion-recovery-boundaries) and [Data Hub recovery](/docs/use/portal/data-hub#recover-progress).

## Interacts With [#interacts-with]

* `Portal` and `BSQAI API`, for configuration, operations and diagnostics.
* `Keycloak` and `OpenFGA`, for tenant membership, functional permissions and resource grants.
* `Temporal`, for durable execution and scheduling.
* `Kubernetes`, for coordinator Jobs, connector Pods and network controls.
* `PostgreSQL`, for application metadata, encrypted credentials and run history.
* `Rook-Ceph` or external S3-compatible storage, for checkpoint backups and customer-configured data paths.