Rook-Ceph

Kubernetes-native S3-compatible object storage, with optional block and shared-file storage, for Foundation.

Agentic Friendly

Component Category

Data and storage / object storage, optional block and file storage

Component Description

Rook-Ceph is the Rook operator running a Ceph cluster on Kubernetes. It provides a RADOS Gateway (RGW) endpoint for S3-compatible object storage and is the platform's primary object storage layer. It can also provide RBD block volumes and CephFS shared file storage when the site uses Ceph for its persistent volume classes.

Why It Is Used

In BullSequana AI Foundation, Rook-Ceph provides a Kubernetes-native, scalable, and operator-managed storage layer for S3-compatible workloads. On sites without another shared-file provider, CephFS supplies shared file storage. It integrates with the existing identity, ingress, and certificate-management components.

Learn More

S3 Access Model

Two S3 access paths are configured on the RGW:

  • Static access keys — used by services and the operator validation procedure. A bootstrap job creates a dedicated RGW user and writes its access key and secret to the rook-ceph-rgw-bootstrap secret in the rook-ceph namespace. Workloads and operators consume those credentials directly.
  • STS / Keycloak-federated access — RGW enables STS. With sts_federation = true in [object_storage.capabilities], Keycloak web-identity tokens can be exchanged for temporary S3 credentials through AssumeRoleWithWebIdentity, subject to the configured role trust and permissions policies. The Rook-Ceph configuration enables this capability by default.

Lakekeeper has a dedicated bucket-scoped STS role for credential vending. Spark storage roles are scoped to tenant prefixes within that bucket. Each tenant's role accepts only web-identity tokens that carry that tenant's Keycloak organization ID in the tenant claim (tenant_id by default), so a token from one tenant cannot assume another tenant's role. The shared bootstrap role KeycloakS3Role denies all S3 access and is not a substitute for a tenant-scoped role. Catalog permissions remain separate from S3 role policies. External S3 providers follow their own supported authentication path, not the platform's RGW federation setup.

For static-key validation and the separate federation checks, see Rook-Ceph RGW S3 access.

Dashboard access

The Ceph dashboard is exposed through Gateway API. An Envoy Gateway SecurityPolicy performs OIDC authentication with Keycloak before the route forwards traffic directly to the Ceph manager dashboard. A dedicated Keycloak client, rgw-client, is created during deployment; the dashboard does not use an OAuth2 Proxy instance. Members of COREAI-ROOK-CEPH-ADMIN-GROUP get the dashboard administrator role, and members of COREAI-ROOK-CEPH-USER-GROUP get read-only.

Operational Notes

  • An orphan-cleaner cronjob runs in the rook-ceph namespace and removes pods that remain bound to non-existent nodes, which would otherwise block component startup after node turnover.
  • While the Rook-Ceph components are enabled, RGW object storage is always deployed. Sites that use external S3 object storage turn both components off (ROOK_CEPH_COMPONENT_ENABLED=false and ROOK_CEPH_CLUSTER_COMPONENT_ENABLED=false); see External S3 object storage.
  • Block (ceph-block) and shared-file (ceph-filesystem) storage classes are deployed only when their mode is enabled. A mode is enabled when [common] rwo_storageclass or rwx_storageclass names its class, or when it is forced on. See Ceph storage modes.
  • Turning a mode off removes its pool, storage class, and CSI driver. An admission policy rejects new claims on the removed class, and the removal stops while any volume still uses it. See Argo CD behavior.
  • TLS for RGW and dashboard endpoints is issued by cert-manager.
  • Before Argo CD applies the Ceph image and key settings, a PreSync job prepares the Ceph daemon keys for rotation. The cluster then moves existing daemon keys to the secure key type ([cephx] daemon_key_generation = 2), which addresses CVE-2025-30156; new clusters start in that state. Keep the generation value monotonic: never lower it.

Deployment notes

Rook-Ceph deploys as two components in the common tier: the Rook v1.20 operator at sync wave 5 and the Ceph v20.2 cluster at sync wave 8, both in the rook-ceph namespace. The cluster runs three MON daemons, one MGR, and one OSD device set with a count of 3, spread across hosts. The object store uses three-way replication for both metadata and data pools. OSDs prefer nodes labelled workload=storage and tolerate the dedicated=storage taint.

Interacts With

  • cert-manager, which issues and renews TLS certificates for RGW and dashboard endpoints.
  • Gateway API and External DNS, which expose the RGW S3 endpoint and the Ceph dashboard.
  • Keycloak, which authenticates dashboard access through Envoy Gateway OIDC and supplies web-identity tokens for the STS-based S3 access path.
  • Reflector, which is used to propagate selected secrets across namespaces during deployment.

On this page