External S3 object storage

Configure and validate customer-managed S3-compatible storage as the platform object store.

Agentic Friendly

BullSequana AI can use a customer-managed S3-compatible service instead of the Rook-Ceph object gateway. The platform renders the endpoint, CA trust, and per-component Kubernetes secrets, while the storage operator owns the accounts, buckets, access policies, availability, and recovery process.

This playbook covers the primary [object_storage] provider. It does not require the optional [model_storage] endpoint and does not migrate existing objects automatically.

The storage operator provides the external-storage configuration through one approved handoff. A BullSequana AI deployment operator maps that information into the protected site configuration and renders the sealed Kubernetes secrets. Portal users and tenant administrators do not configure these credentials.

Understand the storage boundaries

ConfigurationPurposeRequired
[object_storage]Primary store for files, model artifacts, logs, traces, backups, pipelines, and other platform dataYes
[model_storage]Separate, read-only-capable repository used by the direct-S3 KServe runtime profileNo
RWO and RWX storage classesKubernetes persistent volumes for databases and stateful workloadsYes

Selecting external-s3 replaces the platform's object-store endpoint. It does not provide block or shared-file storage. Before disabling the Ceph components, configure working RWO and RWX storage classes from another supported storage provider.

1. Confirm provider compatibility

Collect and test the following information with the storage operator:

  • a private HTTPS endpoint reachable from cluster workloads
  • a browser-reachable DNS name for presigned downloads
  • the endpoint's explicit port, region, and addressing style
  • the CA certificate when HTTPS uses a private trust chain
  • support for the required bucket, object, multipart-upload, and lifecycle operations
  • the provider's server-side encryption and versioning behavior
  • how one service account can access a bucket owned by another account
  • the provider's presigned-request handling, especially Content-Type

The public host is supplied without a scheme. BullSequana AI constructs browser-facing URLs with HTTPS, so its certificate and DNS name must be valid for users as well as reachable from any workload that uses that URL.

Use HTTPS for the object storage endpoint, and set OBJECT_STORAGE_SCHEME=https even when the provider also offers a plain HTTP listener. Private tenant MLflow and application buckets refuse an S3 endpoint that is not HTTPS.

Set the final endpoint before any tenant gets storage. The tenant operator records each tenant's endpoint, region, and provisioner Secret when it first provisions that storage. It refuses a later change, including a move from HTTP to HTTPS or to another port, with StorageBindingImmutable.

2. Prepare accounts and buckets

The named application buckets and their credentials must exist before rendering the platform. The storage operator includes their account-to-bucket mapping in the site handoff. The platform does not create them through Rook ObjectBucketClaim resources.

The default deployment uses these buckets when all corresponding components are enabled:

ConsumerDefault bucket or prefixPurpose
Grafana Lokiloki-chunks, loki-ruler, loki-adminlogs, rules, and Loki administration data
Grafana Tempotempo-tracesdistributed traces
Grafana Mimirmimir-blockslong-term metric blocks
CloudNativePGcnpg-backupsbase backups and WAL archives under cnpg/
Milvusmilvusvector-store data
MLflowmlflowmodel and run artifacts
BSQAI APIdefault, default-artifactsapplication files and artifacts
BSQAI API, Spark jobsspark-job-artifactssource files of Spark jobs submitted through the BSQAI API, read by Spark pods through short-lived presigned URLs; uses the BSQAI API account
Agent sandbox (BSQAI API account)sandbox-workspacesthe Global organization's agent workspaces, when the sandbox is enabled
KServekservemodel-serving artifacts
Managed ingestionmeltano-statetenant/ingestion checkpoint backups, not destination records
Argo Eventsargo-eventsevent payloads and artifacts
Kubeflow Pipelineskubeflow-pipelinescentral pipeline artifacts
Lakekeeperlakekeepergoverned table and warehouse objects

Grafana Mimir is enabled by default. Its mimir-blocks bucket and its MIMIR_S3_* credential pair are therefore required, unless the site sets MIMIR_COMPONENT_ENABLED=false, which also removes the platform's long-term metrics backend. The render fails while an enabled consumer has no credentials.

A tenant's agent workspaces are saved under sandbox/ in its <tenantId>-data bucket. For sandbox-workspaces, the agent sandbox uses the BSQAI API account (LLM_BACKEND_S3_*), which needs read, write, list, and delete access to it. Its reaper connects to the endpoint in OBJECT_STORAGE_PUBLIC_HOST, so that host must also be reachable from the BSQAI API namespace. Set LLM_BACKEND_SANDBOX_BUCKET when the bucket has another name. The BSQAI API does not create or check the bucket, so a missing bucket or permission shows only when a workspace is saved or restored. See Enable and operate agent code sandboxes.

The tenant operator creates a <tenantId>-data bucket per tenant that selects storage or private MLflow; MLflow writes its artifacts under mlflow/. The BSQAI API reaches these buckets with the same provisioner account, rendered into its tenant-data-s3 Secret, for each tenant's library files under library/ and agent workspaces under sandbox/. Its provisioner account needs bucket creation and object read, write, list and delete on *-data buckets, including version listing and deletion for tenant deletion with the Delete policy, and no access to other buckets. Private MLflow uses the same account; see Configure private tenant MLflow.

Kubeflow creates buckets at runtime with its storage account: by default kfp-<slug>-<hash> for each profile's pipeline artifacts, and tenant-derived user-* and space-* buckets. That account needs the provider permissions required to create and manage those prefixed buckets, including PutBucketCORS on user-* and space-*.

Use a dedicated account for each enabled consumer where the provider supports it. This limits the impact of a compromised credential and allows each account to receive only the bucket and object operations that its component needs.

A provider can instead supply one shared account for several or all consumers. That account must have access to every required bucket. The deployment operator maps the same pair into the applicable component credential slots. This is supported, but it creates a broader access boundary.

Shared MLflow access

The mlflow bucket crosses component boundaries. Grant this access on the storage provider before deployment:

AccountCredentialsAccess to mlflow
MLflowMLFLOW_S3_*Owns the artifact store: read, write, list, and delete
KServeKSERVE_RESOURCES_S3_*Reads model artifacts during model loading: list the bucket and read artifacts/*
BSQAI APILLM_BACKEND_S3_*Reads the tokenizer files of models deployed from the store: read artifacts/*

The BSQAI API and Model Installer upload model artifacts through the MLflow server, so the MLflow account writes those objects.

Set [object_storage.capabilities] cross_account to describe how the provider enforces this boundary:

  • policy — bucket policies grant access to the other accounts
  • posix — the service also enforces filesystem ownership or group access
  • none — cross-account sharing is not available

With posix, a bucket policy alone might not be sufficient. Configure the provider's filesystem ownership and group permissions as well, so that the KServe and BSQAI API accounts can read the files the MLflow account creates, including files created after setup. With none, use a reviewed provider-side alternative, such as one credential shared only by the required consumers.

3. Configure the endpoint and capabilities

Put site-specific values in the platform environment file rather than changing the shipped defaults. This example uses a provider that supports bucket policies, SSE-S3, and versioning:

OBJECT_STORAGE_PROVIDER=external-s3
OBJECT_STORAGE_SCHEME=https
OBJECT_STORAGE_HOST=s3.internal.example.com
OBJECT_STORAGE_PORT=443
OBJECT_STORAGE_REGION=eu-west-1
OBJECT_STORAGE_PATH_STYLE=true
OBJECT_STORAGE_PUBLIC_HOST=s3.example.com

# Leave empty when the endpoint chains to a public CA already trusted by the platform.
OBJECT_STORAGE_CA_PEM="<PEM-encoded private CA certificate>"

OBJECT_STORAGE_CAPABILITIES_SSE=true
OBJECT_STORAGE_CAPABILITIES_PRESIGN_SIGNS_CONTENT_TYPE=false
OBJECT_STORAGE_CAPABILITIES_CROSS_ACCOUNT=policy
OBJECT_STORAGE_CAPABILITIES_STS_FEDERATION=false
OBJECT_STORAGE_CAPABILITIES_STS=false
OBJECT_STORAGE_CAPABILITIES_VERSIONING=true
OBJECT_STORAGE_CAPABILITIES_MANAGE_BUCKET_CORS=false

Set the capability values to match the service that was tested; they are not preference flags.

CapabilityMeaning for an external provider
sseEnables SSE-S3 requests only when the provider supports them
presign_signs_content_typeRecords whether the provider validates Content-Type on presigned uploads
cross_accountSelects policy, posix, or none for shared-bucket access
sts_federationMust be false; Keycloak-to-RGW federation is specific to Rook-Ceph
stsKeep false unless the provider is verified to intersect an assumed-role session policy with the account policy
versioningEnables behavior that depends on bucket versioning only when the provider supports it
manage_bucket_corsAllows BullSequana AI to replace the bucket's complete CORS document

Use OBJECT_STORAGE_PATH_STYLE=true when the provider expects https://host/bucket/key. Use false only when wildcard DNS and certificates support virtual-hosted requests such as https://bucket.host/key.

4. Hand off storage credentials

The storage operator provides one site bundle through the approved secret-handoff process. Include:

  • each provider-issued access-key and secret-key pair
  • the account-to-bucket mapping and allowed operations
  • the MLflow account, which writes model artifacts to the shared mlflow bucket
  • the tenant provisioner account limited to *-data buckets, when private tenant MLflow is used
  • any additional policy or filesystem-group assignments used for cross-account access

The BullSequana AI deployment operator stores this bundle through the site's secret-management process and maps it into the protected platform environment. The platform currently uses separate configuration slots because it renders a Kubernetes Secret for each component. It does not require every slot to contain a different provider account.

Components that are disabled need no credential mapping. For the exact deployment variables, see Credential mapping reference.

5. Configure browser CORS

Presigned file downloads originate from the portal in the user's browser. Each affected external bucket must allow GET and HEAD from the portal origin.

A provider-neutral policy has this shape:

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://portal.example.com"],
      "AllowedMethods": ["GET", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag", "Content-Length", "Content-Type", "Content-Disposition", "Content-Range", "Accept-Ranges"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Prefer OBJECT_STORAGE_CAPABILITIES_MANAGE_BUCKET_CORS=false and let the storage operator own this policy. Set it to true only when the storage owner explicitly delegates the complete CORS document to BullSequana AI. PutBucketCors replaces the existing document rather than merging a rule into it. The setting covers default and default-artifacts only.

Buckets created at runtime

The platform creates these buckets after deployment, so neither a policy set before deployment nor manage_bucket_cors reaches them:

BucketCORS
<tenantId>-dataThe platform sets no policy. Apply the policy above after the tenant operator creates the bucket, and repeat this for every new tenant. Do not create the bucket in advance: the operator refuses a bucket it did not create.
user-*, space-*The Kubeflow profile bucket provisioner replaces the policy with GET and HEAD for the Lakekeeper UI origin. If the provider refuses the call, the provisioner logs a warning and continues.
kfp-*The platform sets no policy, and none is needed. The KFP UI reads these artifacts on the server, not in the browser.

Check which rule the provider applies to a new bucket by default. A default rule can allow every origin, which still requires a valid signature but is broader than the platform needs. On a <tenantId>-data bucket, a default rule that does not expose Accept-Ranges also makes the portal download a whole PDF to preview it.

6. Disable unused Ceph components

When the site does not use Rook-Ceph for object, block, or shared-file storage, disable its components:

ROOK_CEPH_COMPONENT_ENABLED=false
ROOK_CEPH_CLUSTER_COMPONENT_ENABLED=false
CEPH_CSI_DRIVERS_COMPONENT_ENABLED=false

Do not disable the CSI drivers until replacement RWO and RWX storage classes are configured and tested. Selecting external-s3 alone does not create those classes.

7. Validate and render

Before rendering, test each writable account against every bucket it is expected to use. When the provider supplies one shared account, repeat the bucket checks with that account:

export AWS_ACCESS_KEY_ID=<component-access-key>
export AWS_SECRET_ACCESS_KEY=<component-secret-key>
export AWS_DEFAULT_REGION=eu-west-1
export S3_ENDPOINT=https://s3.internal.example.com:443

aws --endpoint-url "$S3_ENDPOINT" s3api head-bucket --bucket <component-bucket>
aws --endpoint-url "$S3_ENDPOINT" s3 cp ./s3-smoke-test.txt "s3://<component-bucket>/s3-smoke-test.txt"
aws --endpoint-url "$S3_ENDPOINT" s3 cp "s3://<component-bucket>/s3-smoke-test.txt" ./s3-smoke-test.downloaded.txt
aws --endpoint-url "$S3_ENDPOINT" s3 rm "s3://<component-bucket>/s3-smoke-test.txt"

For a read-only account, replace the write and delete operations with head-object and a download of a known test object. Also test the MLflow bucket with the MLflow writer and the KServe reader credentials.

Render the complete desired state:

bsqai apply all -r

Rendering fails closed when the provider, host, scheme, cross-account mode, STS federation setting, or required component credentials are inconsistent. This includes the tenant operator's BSQAI_TENANT_OPERATOR_S3_* pair unless the site manages the provisioner Secret itself. Treat those errors as configuration failures; do not replace missing external credentials with generated placeholder values.

Review the generated manifests before pushing them. Confirm that:

  • endpoints include the intended explicit port
  • the public hostname is correct for presigned URLs
  • private CA material is mounted where required
  • external credentials are rendered as Sealed Secrets
  • Rook bucket and user resources are absent for external-S3 consumers
  • disabled Ceph applications are absent from the generated app-of-apps output

8. Verify after deployment

Run an end-to-end check for every enabled storage path:

  1. Upload and download a file through the portal, in the Global organization and in a tenant, then verify the browser reports no CORS error.
  2. Process a file for RAG and confirm that Milvus search returns it.
  3. Create and verify a CloudNativePG backup in cnpg-backups.
  4. Confirm Loki logs, Tempo traces and Mimir metrics continue to appear in Grafana.
  5. Import or deploy a model and verify the MLflow writer and KServe reader paths.
  6. Run a Kubeflow pipeline and confirm its artifacts, if Kubeflow is enabled.
  7. Verify managed-ingestion checkpoint backups and an Argo Events artifact flow when enabled. Separately test any data ingestion connection that uses this object store as a source or destination, then verify its ingestion result.
  8. If the agent sandbox is enabled, leave a sandbox session of a tenant idle until the reaper stops it, confirm that its workspace objects appear under sandbox/<tenantId>/<chat-id>/ in the tenant's <tenantId>-data bucket, and confirm that the next message in the chat restores the files.

Inspect provider audit logs for denied requests, unexpected bucket access, TLS failures, or requests sent to the public endpoint when the private endpoint was expected.

Migration and rollback

Changing OBJECT_STORAGE_PROVIDER changes where the platform looks for objects; it does not copy data from Rook-Ceph or rewrite existing object references.

Before switching an existing deployment:

  1. stop or quiesce writers
  2. copy every required bucket, object version, and metadata record
  3. validate object counts, sizes, checksums, retention, and restore procedures
  4. update the endpoint, capabilities, credentials, and Ceph component settings together
  5. render and review the complete desired state
  6. keep the previous store read-only until application-level validation succeeds

Rollback requires the previous endpoints, credentials, manifests, and object contents to remain available. Do not delete or reuse the old buckets as part of the initial cutover.

Tempo verifies the endpoint certificate, like the other storage consumers. Earlier releases skipped this check, so an existing site can have a problem that only appears at upgrade. Before you sync the upgrade, check that:

  • a private or self-signed certificate is trusted through OBJECT_STORAGE_CA_PEM, or through the Secret and key named by OBJECT_STORAGE_CA_SECRET and OBJECT_STORAGE_CA_KEY, which must exist in the Tempo namespace
  • the certificate has a subject alternative name that matches OBJECT_STORAGE_HOST

Otherwise Tempo stops at startup with an x509 error.

The managed-ingestion backup identity is separate from the credentials of data ingestion connections. Source and destination credentials belong to their connections and need the appropriate native bucket permissions. See managed ingestion.

Credential mapping reference

This mapping is performed by the BullSequana AI deployment operator. Set the pair for every enabled S3 consumer in the protected platform environment. A dedicated provider account is recommended for each consumer. When the provider supplies one shared account, map the same pair into each required slot.

GRAFANA_LOKI_S3_ACCESS_KEY=<access-key>
GRAFANA_LOKI_S3_SECRET_KEY=<secret-key>
TEMPO_S3_ACCESS_KEY=<access-key>
TEMPO_S3_SECRET_KEY=<secret-key>
MIMIR_S3_ACCESS_KEY=<access-key>
MIMIR_S3_SECRET_KEY=<secret-key>
MILVUS_S3_ACCESS_KEY=<access-key>
MILVUS_S3_SECRET_KEY=<secret-key>
MLFLOW_S3_ACCESS_KEY=<access-key>
MLFLOW_S3_SECRET_KEY=<secret-key>
KSERVE_RESOURCES_S3_ACCESS_KEY=<access-key>
KSERVE_RESOURCES_S3_SECRET_KEY=<secret-key>
MELTANO_S3_ACCESS_KEY=<access-key>
MELTANO_S3_SECRET_KEY=<secret-key>
ARGO_EVENTS_S3_ACCESS_KEY=<access-key>
ARGO_EVENTS_S3_SECRET_KEY=<secret-key>
KUBEFLOW_S3_ACCESS_KEY=<access-key>
KUBEFLOW_S3_SECRET_KEY=<secret-key>
LAKEKEEPER_S3_ACCESS_KEY=<access-key>
LAKEKEEPER_S3_SECRET_KEY=<secret-key>
LLM_BACKEND_S3_ACCESS_KEY=<access-key>
LLM_BACKEND_S3_SECRET_KEY=<secret-key>
CNPG_BACKUPS_ACCESS_KEY_ID=<access-key>
CNPG_BACKUPS_ACCESS_SECRET_KEY=<secret-key>
BSQAI_TENANT_OPERATOR_S3_ACCESS_KEY=<tenant-provisioner-access-key>
BSQAI_TENANT_OPERATOR_S3_SECRET_KEY=<tenant-provisioner-secret-key>

The tenant operator pair is the provisioner account limited to *-data buckets. The platform renders it as the tenant-storage-provisioner Secret. A site that manages that Secret outside the platform sets BSQAI_TENANT_OPERATOR_STORAGE_PROVISIONER_SECRET_NAME to its name instead; the pair is then not required.

The Model Installer has two additional configuration fields for the mlflow bucket. Set them to the MLflow account's pair, the same values as MLFLOW_S3_*.

LLM_BACKEND_MLFLOW_S3_ACCESS_KEY_ID=<mlflow-access-key>
LLM_BACKEND_MLFLOW_S3_SECRET_ACCESS_KEY=<mlflow-secret-key>

Do not commit these values. The platform seals the rendered Kubernetes secrets, but the source credentials remain in the site's secret-management process.

Troubleshooting

  • AccessDenied during rendering or startup — verify the mapped credential and its bucket policy for the affected component.
  • TLS verification failure — supply the private CA through OBJECT_STORAGE_CA_PEM or fix the endpoint certificate chain. A certificate with only a common name fails with x509: certificate relies on legacy Common Name field; reissue it with a subject alternative name.
  • Requests use the wrong port — set OBJECT_STORAGE_PORT explicitly and verify the rendered endpoint.
  • Browser download fails while backend access works — verify OBJECT_STORAGE_PUBLIC_HOST, its certificate, and the bucket CORS policy, including <tenantId>-data for a tenant.
  • Presigned upload fails only on one provider — verify OBJECT_STORAGE_CAPABILITIES_PRESIGN_SIGNS_CONTENT_TYPE against that provider's signing behavior.
  • MLflow import succeeds but model loading fails — test the mlflow bucket with the KServe credential and review cross-account enforcement.
  • A tenant with private MLflow reports StorageProviderConfigurationMissing — the tenant operator has no provisioner Secret; set the BSQAI_TENANT_OPERATOR_S3_* pair or the externally managed Secret name and render again.
  • A tenant with private MLflow reports InvalidStorageBinding — this can mean that the tenant's S3 endpoint is not HTTPS or embeds credentials, that the tenant has no storage binding, or that its MLflow S3 credential Secret has the wrong name or owner. Check the endpoint in the tenant's status.storage.endpoint. Setting OBJECT_STORAGE_SCHEME=https does not repair a tenant that already has storage on an HTTP endpoint; it causes StorageBindingImmutable.
  • A tenant reports StorageBindingImmutable — the configuration no longer matches the binding recorded when the tenant's storage was provisioned. After a change to the endpoint, region, or provisioner Secret, every tenant with storage reports this. Restore the previous values and render again. Moving a provisioned tenant to another endpoint is an explicit migration; the platform does not move the binding or the data. Do not delete the <tenantId>-storage-binding ConfigMap to force the change.
  • Private MLflow cannot reach an endpoint on a port other than 443 — its egress policy has a separate port setting. Set BSQAI_TENANT_OPERATOR_MLFLOW_STORAGE_PORT to the endpoint port and render again. See External S3.
  • PostgreSQL backups fail — verify the cnpg-backups bucket, the two CNPG_BACKUPS_* credentials, retention support, and CA trust.

On this page