Working with data
Choose the right BullSequana AI data path for ingestion, processing, governance, analytics, and AI knowledge.
BullSequana AI provides several data paths rather than one universal pipeline. Start from the outcome, then choose the managed service and storage boundary that fit the source, latency, governance, and consumer.
Choose the data path
| Goal | Start here | Main platform services |
|---|---|---|
| Load data from an application, database, file, or API on a schedule | Create source and destination connections and an ingestion in Data Hub | Managed ingestion, BSQAI API, Temporal and customer-owned connectors |
| Publish and consume event streams | Define topics, producers, consumers, retention, and ownership | Kafka |
| Explore data or build repeatable processing | Open a managed development environment, then package the work as a pipeline | Developer Workspace, Kubeflow Pipelines, Spark Operator |
| Discover governed tables through an API | Query the Data catalog for tables visible to your identity | Authorization service, metastore, and optional metadata enrichment |
| Build governed analytical tables | Create the approved project and warehouse boundary before writing Iceberg tables | Lakekeeper, Spark, S3-compatible object storage |
| Build dashboards and explore governed datasets | Connect an approved data source and publish through the correct tenant or team | Superset |
| Turn documents into knowledge for AI | Upload through the portal library and wait for indexing to complete | Files and RAG, Docling, Milvus, object storage |
| Track experiments and register models | Log the run, parameters, metrics, and artifacts before model hand-off | MLflow |
For a guided data-engineering journey, use Build a data and ML workflow.
Understand where data lives
| Data | System of record in the platform |
|---|---|
| Files, pipeline artifacts, model artifacts, and table data | Configured S3-compatible object storage: in-cluster Rook-Ceph or external S3 |
| Application metadata and transactional state | PostgreSQL through CloudNativePG-managed services |
| Iceberg catalog, projects, and warehouse metadata | Lakekeeper, with table objects in S3-compatible storage |
| Embeddings and retrieval payloads | Milvus |
| Event streams | Kafka |
| Experiment runs, metrics, and registered-model metadata | MLflow, with artifacts in configured object storage |
Do not build an endpoint by concatenating a host and path. Use the endpoint, addressing style, TLS, public-presign host, and capability settings defined by the platform's object-storage configuration.
Typical flow
These arrows describe configured workflows, not automatic integrations. Ingesting records does not register Iceberg tables or index documents for RAG; those require the corresponding catalog/write or Files upload steps. Superset queries a configured SQL data source.
The exact path can be batch, streaming, or interactive. Production work should make inputs, outputs, schedules, retry behavior, data quality, and ownership explicit rather than relying on a notebook or one-off transfer.
Managed ingestion runs through the BSQAI API and Temporal, independently of Kubeflow Pipelines. Its checkpoint backups protect progress, not destination data. See Data Hub for connection setup and run verification.
Tenant and team boundaries
Keycloak Organizations establish tenant membership. Inside a tenant, team and personal workspaces define who can operate notebooks, pipelines, artifacts, and governed data assets. In the Data stack:
- a Keycloak Organization maps to a Lakekeeper project with a default warehouse
- personal and shared-space catalog namespaces live inside the tenant's default warehouse, rather than creating a warehouse for every bucket
- the Kubeflow profile reconciler creates the user's personal bucket, ensures buckets for the user's space memberships, and ensures the corresponding Lakekeeper namespaces
- deleting a user Profile archives only its personal bucket; shared-space buckets persist with the space
- Tuples records ownership and parent-space relationships for spaces, buckets, datasets, and tables
- service credentials are injected into managed workloads rather than copied into notebooks or source code
Read Multi-tenancy and tiered RBAC before sharing data or promoting a workflow.
Production checklist
Before a workflow becomes operational, confirm:
- the source, accountable owner, classification, and permitted purpose are recorded
- the tenant and team access boundary is tested with an allowed and a denied user
- schemas, freshness, retention, and data-quality expectations are measurable
- credentials come from the platform's approved secret path
- another authorized user can reproduce the processing from versioned code or a pipeline definition
- failures, retries, artifacts, logs, and downstream impact are visible
- backup, restore, and deletion behavior match the system of record
Use Data for the capability map and Data component catalog when you need implementation detail.