Kubeflow

Pipelines, notebooks, and experiment management for ML, DL, and GenAI workflows.

Agentic Friendly

Component Category

Machine learning / pipelines and notebooks

Component Description

Kubeflow is a Kubernetes-native platform for building and managing ML, deep learning, and GenAI workflows. In BullSequana AI it provides Kubeflow Pipelines for orchestrating experiments, a notebook server manager (Developer Workspace) for interactive development environments (JupyterLab, VSCode), and a profile system for multi-tenant namespace isolation. It is deployed as raw Kustomize manifests (not a Helm chart), based on upstream tag 26.03 with local overlays.

Why It Is Used

In BullSequana AI, Kubeflow provides the experimentation and pipeline execution layer for the Data tier. It gives data scientists and AI engineers a structured environment to build, run, and track ML, DL, and GenAI workflows with pipeline versioning, artifact tracking in S3-compatible object storage, and per-profile namespace isolation. Kubeflow complements MLflow (experiment tracking, reached through the BSQAI API) and Argo Workflows (general-purpose automation) by focusing specifically on pipeline orchestration and notebook-based development.

Learn More

Deployment notes

Kubeflow deploys in the proai tier at sync wave 6. It depends on two components that must be running first:

  • Kyverno (sync wave 5) — runs the ClusterPolicy generate and mutate rules that create the profile access tiers, platform PodDefaults and per-user Spark identity settings
  • Metacontroller (sync wave 5) — provides the DecoratorController CRD used by Kubeflow Pipelines

Kubeflow uses Kustomize format instead of Helm. The chart_name field is intentionally empty, so ArgoCD applies the raw manifests from Git instead of pulling a chart from the OCI registry.

By default each profile namespace gets its own pipeline-artifact bucket, and that profile's KFP launcher and artifact repository point at it; the shared kubeflow-pipelines bucket remains for the central KFP server. Object-store credentials are still shared, so per-profile buckets separate artifact layout, not credentials. [s3] bucket_per_profile = false in the Kubeflow component variables reverts to the single shared bucket.

When a user Profile is reconciled, the profile bucket provisioner creates a personal user-* bucket and ensures a space-* bucket for each of that user's space memberships. It records ownership and parent-space relationships through Tuples and ensures personal and shared-space Lakekeeper namespaces inside the tenant's one shared warehouse. Deleting the user Profile archives only the personal bucket; space buckets persist because they belong to the space. The provisioner also creates a personal Spark workload ServiceAccount (u-<keycloak user id>) for each profile editor and administrator and registers it in Lakekeeper. Object storage can be Rook-Ceph RGW or the configured external S3-compatible provider.

Authentication is handled through an internal OAuth2 Proxy instance configured against Keycloak, with access controlled through the COREAI-KUBEFLOW-ADMIN-GROUP and COREAI-KUBEFLOW-USERS-GROUP groups.

Kubeflow also deploys an Istio 1.29.8 service-mesh layer, in Ambient mode by default. Istio CNI and ztunnel provide the node-level data plane, while istiod manages policy and a dedicated ingress gateway receives the public Kubeflow route. Sites whose CNI cannot run Ambient, such as Scaleway's managed Cilium, set [istio] mode = "sidecar" in the Kubeflow component variables. Istio RequestAuthentication and AuthorizationPolicy resources validate Keycloak JWTs and restrict access between the ingress gateway, OAuth2 Proxy, Kubeflow services, and profile namespaces.

Custom container images are built for the central dashboard, Jupyter web app, and KFP frontend, stored in the platform registry.

Kyverno generates the platform PodDefaults in every profile namespace, and every pod there receives them automatically: the registry pull secret, the CA bundle trust, the Lakekeeper tenant context (LAKEKEEPER_URL and LAKEKEEPER_PROJECT_ID, only in namespaces linked to a tenant's Data services) and PLATFORM_API_URL, which the Spark helper and MLflow clients use to reach the BSQAI API.

The notebook spawner offers the images in the [notebooks] jupyterlab and vscode lists of the Kubeflow component variables. Each entry is a path under the platform registry's kubeflow/ repository, and the first entry is the preselected image. By default both lists hold only the Python 3.12 image. KUBEFLOW_NOTEBOOKS_JUPYTERLAB and KUBEFLOW_NOTEBOOKS_VSCODE take a JSON list and replace the whole list, so include every image the spawner should offer. For example, an amd64 site that offers the Databricks Runtime 17.3 LTS image as well:

KUBEFLOW_NOTEBOOKS_JUPYTERLAB='["notebooks/python/jupyterlab:python3.12", "notebooks/databricks-runtime/standard/jupyterlab:17.3-lts"]'

Kubeflow Pipelines runs version 2.17.2. The KFP UI reads artifact previews from the platform object storage with the shared Kubeflow S3 credentials, not from AWS S3. The platform does not deploy the KFP v1 metadata writer (kfp-metadata-writer), so pods of v1 pipelines do not record ML Metadata.

Kubeflow Pipelines preloads two example pipelines, [Tutorial] Data passing between components and [Tutorial] Control structures. They replace the upstream samples and run on the platform notebook image, so they need no access to Docker Hub or PyPI. On a cluster that already loaded the upstream tutorials, those stay in the pipeline list until an operator deletes them.

Interacts With

  • Kyverno, which generates and mutates the profile access tiers, platform PodDefaults and Spark identity settings.
  • Metacontroller, which provides the DecoratorController CRD required by Kubeflow Pipelines.
  • Keycloak, which provides SSO authentication through an internal OAuth2 Proxy instance.
  • Istio, which provides the Ambient data plane, ingress boundary, JWT validation, and service authorization policies.
  • Rook-Ceph or external S3-compatible storage, which stores pipeline artifacts and user data.
  • Tuples and OpenFGA, which record and authorize personal and shared-space object-storage relationships.
  • Lakekeeper and Spark Operator, which support catalog-backed distributed data processing in profile namespaces.
  • BSQAI API, which serves Spark management (/spark) and MLflow (/v1/mlflow) to notebooks and pipeline steps; dev environments and pipelines have no direct MLflow access.
  • trust-manager, which distributes CA trust bundles into Kubeflow namespaces for TLS verification.

On this page