Data

Connect, govern, process, analyze, and operationalize data on BullSequana AI.

Agentic Friendly

Data is the product domain for moving information from source systems into governed pipelines, workspaces, models, analytics, and AI experiences. It connects data engineering and data science without requiring users to operate the underlying platform components.

Choose an outcome

Capability map

Integrate and move data

Data integration connects applications, databases, object stores, and external services to governed platform workflows. Connectors define the source and destination; orchestration controls when work runs; operational signals show whether transfers are healthy.

In the Portal, the Data Ingestion group of Studio configures connections, discovers streams, schedules ingestions and shows runs; see Data ingestion in Studio. The managed-ingestion reference explains the BSQAI API, Temporal and connector runtime, including Meltano/Singer integration.

Engineer and orchestrate data

Lakehouse and pipeline capabilities organize catalog-backed tables, processing jobs, artifacts, schedules, and dependencies. Developer Workspace pipelines provide the practical path for building and operating reusable workflows.

Process data at scale

Apache Spark runs distributed processing in the user's profile namespace. Compute in Studio creates Spark clusters and runs one-off PySpark jobs; dev environments connect to a running cluster with the platform helper. The BSQAI API applies Spark roles and per-user limits to every path. See Run Spark workloads.

Govern and discover data

The data catalog combines authorized table discovery, schema, and optional enrichment metadata. Users search it in Studio under Discover > Data Catalog, and applications call it through the BSQAI API.

Catalog and authorization capabilities describe schemas, ownership, permissions, lineage, and the tenant or team allowed to use an asset. Multi-tenancy and tiered RBAC explains how those boundaries apply across Data. Metastore explains how tables are cataloged and how access is granted.

Develop models and experiments

Developer Workspace provides notebook-first and Python-first environments. MLflow records experiments, artifacts, and model registrations so that development work can move into governed AI serving without losing its history. Notebooks and pipelines reach MLflow through the BSQAI API.

Start with track experiments and register models.

Stream and analyze

Streaming moves events through Kafka-backed paths. Analytics and BI turn analytical data into dashboards and operational views in Superset. These paths share access-control, observability, and storage responsibilities with batch workflows.

Typical data-to-production flow

  1. Identify the source, owner, classification, and expected refresh behavior.
  2. Choose a connector or approved ingestion path.
  3. Develop transformations or models in a managed workspace.
  4. Package the work as a repeatable pipeline with artifacts and parameters.
  5. Track experiments and register models when machine learning is involved.
  6. Apply tenant, team, catalog, and data-access policies before production use.
  7. Monitor freshness, failures, consumption, and downstream impact.

Architecture and implementation

On this page