Spark Operator
Kubernetes operator for tenant-scoped Apache Spark applications.
Component Category
Distributed data processing
Component Description
Spark Kubernetes Operator reconciles SparkApplication and SparkCluster resources into Apache Spark driver, master, worker and executor workloads on Kubernetes. In BullSequana AI, the BSQAI API creates these resources in Kubeflow profile namespaces, so each user runs Spark without operating a separate Spark installation.
Why It Is Used
In BullSequana AI, Spark supplies distributed processing for data preparation, lakehouse tables, and machine-learning pipelines. The operator makes those workloads declarative and compatible with Kubernetes resource controls.
Learn More
Deployment notes
BullSequana AI deploys Apache Spark Kubernetes Operator 1.0.0 from chart 1.8.0 at sync wave 5, with a single replica, images mirrored in the private registry, a cluster-wide watch, and operator metrics on port 19090. Spark clusters and jobs run Apache Spark 4.0.4 from the platform's Spark runtime image; users cannot choose another version.
The BSQAI API creates SparkCluster resources for clusters, and SparkApplication resources for Compute jobs, in the caller's profile namespace. It requires the Spark Operator or Spark Admin role, a writable profile membership, and the per-user limits described in Run Spark workloads.
Spark pods run as a per-user workload ServiceAccount, u- followed by the user's Keycloak ID, which the profile bucket provisioner creates in the profile namespace. A ValidatingAdmissionPolicy and Kyverno policies keep that identity boundary: a pod can run only as its user's own identity, it receives only the platform PodDefaults, and it gets a projected Lakekeeper token. The ValidatingAdmissionPolicy runs inside the Kubernetes API server, so the identity boundary holds even when the Kyverno webhook is unavailable.
The optional S3 credentials broker vends tenant-scoped object-storage credentials to Spark pods that opt in. It is off by default and requires Rook-Ceph RGW with STS enabled in the tenant operator configuration. Without it, Spark pods have no object-storage credentials.
Interacts With
BSQAI API, which creates and deletes Spark clusters and jobs for the Portal, notebooks and pipeline steps.Lakekeeper, which supplies the Iceberg REST Catalog and authenticates per-user Spark pods.Kyverno, which injects the per-user identity settings into Spark pods.Rook-Cephor external S3-compatible storage, which stores lakehouse data and job artifacts.Kubeflow, whose profile namespaces host Spark workloads and whose provisioner creates the per-user Spark ServiceAccounts.GrafanaandPrometheus, which expose operator and workload telemetry.