Reference architecture

A high-level view of how Data services fit together.

Agentic Friendly

Data groups the platform services that focus on ingestion, discovery, streaming, analytical processing, and business intelligence.

Architecture view

Managed ingestion

The BSQAI API applies tenant authorization through Keycloak and OpenFGA. Source and destination credentials remain separate. The coordinator starts the separate source and destination Pods. Records flow through it, not through Temporal history. The meltano deployment component supplies this execution environment; connector images may package MeltanoHub/Singer taps or targets, but the coordinator does not run the Meltano CLI. Use the managed-ingestion reference for checkpoint and backup responsibilities.

Processing, governance and analytics

The ingestion control plane lives in the BSQAI API; record transfer runs between isolated source and destination containers through a coordinator. MeltanoHub/Singer connectors can be packaged for this runtime. Object storage and PostgreSQL are example destinations, not a fixed destination catalogue. See managed ingestion for checkpoint storage, credentials and execution boundaries.

The BSQAI API is the control plane for Spark and the data catalog. It creates SparkCluster and SparkApplication resources in the caller's profile namespace after it checks the caller's Spark role, profile membership and per-user limits. Spark pods run as the user's personal workload identity, which Lakekeeper authorizes with a projected ServiceAccount token. Spark sessions started from a notebook kernel or a pipeline step run their driver as the profile's shared identity and attach no Iceberg catalog. Notebook code can still open the catalog directly with the user's own sign-in token, as Metastore shows. The S3 credentials broker, which vends tenant-scoped object-storage credentials to Spark pods, is off by default.

Catalog discovery forwards the caller's own token to Tuples and Lakekeeper. Dev environments and pipelines reach MLflow only through the BSQAI API. Kafka carries the tenant authorization audit topics and the Milvus write-ahead log; another producer or consumer needs its own KafkaTopic, KafkaUser and network access, created by the platform operator.

Data domains

DomainMain roleExample components
Business intelligenceExplore and visualize analytical dataSuperset
Data ingestion and ETLMove and synchronize data across systemsStudio Data Ingestion, BSQAI API, Temporal and connector containers
Data discoveryFind governed tables and their grantsStudio Data Catalog, BSQAI API, Tuples, Lakekeeper
Event streamingCarry and distribute streaming eventsKafka
ML pipelines and notebooksBuild, run, and track ML/DL/GenAI workflowsKubeflow, MLflow through the BSQAI API
Distributed computeRun PySpark jobs and Spark clustersStudio Compute, BSQAI API, Spark Operator
LakehouseCatalog and process governed Iceberg tablesLakekeeper, Spark Operator
Tenant data servicesAuthorize Data resources per tenant and publish audit eventsTuples

Design intent

Data gives BullSequana AI a dedicated data and analytics layer that complements the Foundation and AI tiers. It allows the platform to support AI systems that rely on richer data movement, event flows, analytical querying, and business reporting.

On this page