Reference architecture
A high-level view of how Data services fit together.
Data groups the platform services that focus on ingestion, discovery, streaming, analytical processing, and business intelligence.
Architecture view
Managed ingestion
The BSQAI API applies tenant authorization through Keycloak and OpenFGA. Source and destination credentials remain separate. The coordinator starts the separate source and destination Pods. Records flow through it, not through Temporal history. The meltano deployment component supplies this execution environment; connector images may package MeltanoHub/Singer taps or targets, but the coordinator does not run the Meltano CLI. Use the managed-ingestion reference for checkpoint and backup responsibilities.
Processing, governance and analytics
The ingestion control plane lives in the BSQAI API; record transfer runs between isolated source and destination containers through a coordinator. MeltanoHub/Singer connectors can be packaged for this runtime. Object storage and PostgreSQL are example destinations, not a fixed destination catalogue. See managed ingestion for checkpoint storage, credentials and execution boundaries.
The BSQAI API is the control plane for Spark and the data catalog. It creates SparkCluster and SparkApplication resources in the caller's profile namespace after it checks the caller's Spark role, profile membership and per-user limits. Spark pods run as the user's personal workload identity, which Lakekeeper authorizes with a projected ServiceAccount token. Spark sessions started from a notebook kernel or a pipeline step run their driver as the profile's shared identity and attach no Iceberg catalog. Notebook code can still open the catalog directly with the user's own sign-in token, as Metastore shows. The S3 credentials broker, which vends tenant-scoped object-storage credentials to Spark pods, is off by default.
Catalog discovery forwards the caller's own token to Tuples and Lakekeeper. Dev environments and pipelines reach MLflow only through the BSQAI API. Kafka carries the tenant authorization audit topics and the Milvus write-ahead log; another producer or consumer needs its own KafkaTopic, KafkaUser and network access, created by the platform operator.
Data domains
| Domain | Main role | Example components |
|---|---|---|
| Business intelligence | Explore and visualize analytical data | Superset |
| Data ingestion and ETL | Move and synchronize data across systems | Studio Data Ingestion, BSQAI API, Temporal and connector containers |
| Data discovery | Find governed tables and their grants | Studio Data Catalog, BSQAI API, Tuples, Lakekeeper |
| Event streaming | Carry and distribute streaming events | Kafka |
| ML pipelines and notebooks | Build, run, and track ML/DL/GenAI workflows | Kubeflow, MLflow through the BSQAI API |
| Distributed compute | Run PySpark jobs and Spark clusters | Studio Compute, BSQAI API, Spark Operator |
| Lakehouse | Catalog and process governed Iceberg tables | Lakekeeper, Spark Operator |
| Tenant data services | Authorize Data resources per tenant and publish audit events | Tuples |
Design intent
Data gives BullSequana AI a dedicated data and analytics layer that complements the Foundation and AI tiers. It allows the platform to support AI systems that rely on richer data movement, event flows, analytical querying, and business reporting.