External S3 object storage
Configure and validate customer-managed S3-compatible storage as the platform object store.
BullSequana AI can use a customer-managed S3-compatible service instead of the Rook-Ceph object gateway. The platform renders the endpoint, CA trust, and per-component Kubernetes secrets, while the storage operator owns the accounts, buckets, access policies, availability, and recovery process.
This playbook covers the primary [object_storage] provider. It does not require the optional [model_storage] endpoint and does not migrate existing objects automatically.
The storage operator provides the external-storage configuration through one approved handoff. A BullSequana AI deployment operator maps that information into the protected site configuration and renders the sealed Kubernetes secrets. Portal users and tenant administrators do not configure these credentials.
Understand the storage boundaries
| Configuration | Purpose | Required |
|---|---|---|
[object_storage] | Primary store for files, model artifacts, logs, traces, backups, pipelines, and other platform data | Yes |
[model_storage] | Separate, read-only-capable repository used by the direct-S3 KServe runtime profile | No |
| RWO and RWX storage classes | Kubernetes persistent volumes for databases and stateful workloads | Yes |
Selecting external-s3 replaces the platform's object-store endpoint. It does not provide block or shared-file storage. Before disabling the Ceph components, configure working RWO and RWX storage classes from another supported storage provider.
1. Confirm provider compatibility
Collect and test the following information with the storage operator:
- a private HTTPS endpoint reachable from cluster workloads
- a browser-reachable DNS name for presigned downloads
- the endpoint's explicit port, region, and addressing style
- the CA certificate when HTTPS uses a private trust chain
- support for the required bucket, object, multipart-upload, and lifecycle operations
- the provider's server-side encryption and versioning behavior
- how one service account can access a bucket owned by another account
- the provider's presigned-request handling, especially
Content-Type
The public host is supplied without a scheme. BullSequana AI constructs browser-facing URLs with HTTPS, so its certificate and DNS name must be valid for users as well as reachable from any workload that uses that URL.
Use HTTPS for the object storage endpoint, and set OBJECT_STORAGE_SCHEME=https even when the provider also offers a plain HTTP listener. Private tenant MLflow and application buckets refuse an S3 endpoint that is not HTTPS.
Set the final endpoint before any tenant gets storage. The tenant operator records each tenant's endpoint, region, and provisioner Secret when it first provisions that storage. It refuses a later change, including a move from HTTP to HTTPS or to another port, with StorageBindingImmutable.
2. Prepare accounts and buckets
The named application buckets and their credentials must exist before rendering the platform. The storage operator includes their account-to-bucket mapping in the site handoff. The platform does not create them through Rook ObjectBucketClaim resources.
The default deployment uses these buckets when all corresponding components are enabled:
| Consumer | Default bucket or prefix | Purpose |
|---|---|---|
| Grafana Loki | loki-chunks, loki-ruler, loki-admin | logs, rules, and Loki administration data |
| Grafana Tempo | tempo-traces | distributed traces |
| Grafana Mimir | mimir-blocks | long-term metric blocks |
| CloudNativePG | cnpg-backups | base backups and WAL archives under cnpg/ |
| Milvus | milvus | vector-store data |
| MLflow | mlflow | model and run artifacts |
| BSQAI API | default, default-artifacts | application files and artifacts |
| BSQAI API, Spark jobs | spark-job-artifacts | source files of Spark jobs submitted through the BSQAI API, read by Spark pods through short-lived presigned URLs; uses the BSQAI API account |
| Agent sandbox (BSQAI API account) | sandbox-workspaces | the Global organization's agent workspaces, when the sandbox is enabled |
| KServe | kserve | model-serving artifacts |
| Managed ingestion | meltano-state | tenant/ingestion checkpoint backups, not destination records |
| Argo Events | argo-events | event payloads and artifacts |
| Kubeflow Pipelines | kubeflow-pipelines | central pipeline artifacts |
| Lakekeeper | lakekeeper | governed table and warehouse objects |
Grafana Mimir is enabled by default. Its mimir-blocks bucket and its MIMIR_S3_* credential pair are therefore required, unless the site sets MIMIR_COMPONENT_ENABLED=false, which also removes the platform's long-term metrics backend. The render fails while an enabled consumer has no credentials.
A tenant's agent workspaces are saved under sandbox/ in its <tenantId>-data bucket. For sandbox-workspaces, the agent sandbox uses the BSQAI API account (LLM_BACKEND_S3_*), which needs read, write, list, and delete access to it. Its reaper connects to the endpoint in OBJECT_STORAGE_PUBLIC_HOST, so that host must also be reachable from the BSQAI API namespace. Set LLM_BACKEND_SANDBOX_BUCKET when the bucket has another name. The BSQAI API does not create or check the bucket, so a missing bucket or permission shows only when a workspace is saved or restored. See Enable and operate agent code sandboxes.
The tenant operator creates a <tenantId>-data bucket per tenant that selects storage or private MLflow; MLflow writes its artifacts under mlflow/. The BSQAI API reaches these buckets with the same provisioner account, rendered into its tenant-data-s3 Secret, for each tenant's library files under library/ and agent workspaces under sandbox/. Its provisioner account needs bucket creation and object read, write, list and delete on *-data buckets, including version listing and deletion for tenant deletion with the Delete policy, and no access to other buckets. Private MLflow uses the same account; see Configure private tenant MLflow.
Kubeflow creates buckets at runtime with its storage account: by default kfp-<slug>-<hash> for each profile's pipeline artifacts, and tenant-derived user-* and space-* buckets. That account needs the provider permissions required to create and manage those prefixed buckets, including PutBucketCORS on user-* and space-*.
Use a dedicated account for each enabled consumer where the provider supports it. This limits the impact of a compromised credential and allows each account to receive only the bucket and object operations that its component needs.
A provider can instead supply one shared account for several or all consumers. That account must have access to every required bucket. The deployment operator maps the same pair into the applicable component credential slots. This is supported, but it creates a broader access boundary.
Shared MLflow access
The mlflow bucket crosses component boundaries. Grant this access on the storage provider before deployment:
| Account | Credentials | Access to mlflow |
|---|---|---|
| MLflow | MLFLOW_S3_* | Owns the artifact store: read, write, list, and delete |
| KServe | KSERVE_RESOURCES_S3_* | Reads model artifacts during model loading: list the bucket and read artifacts/* |
| BSQAI API | LLM_BACKEND_S3_* | Reads the tokenizer files of models deployed from the store: read artifacts/* |
The BSQAI API and Model Installer upload model artifacts through the MLflow server, so the MLflow account writes those objects.
Set [object_storage.capabilities] cross_account to describe how the provider enforces this boundary:
policy— bucket policies grant access to the other accountsposix— the service also enforces filesystem ownership or group accessnone— cross-account sharing is not available
With posix, a bucket policy alone might not be sufficient. Configure the provider's filesystem ownership and group permissions as well, so that the KServe and BSQAI API accounts can read the files the MLflow account creates, including files created after setup. With none, use a reviewed provider-side alternative, such as one credential shared only by the required consumers.
3. Configure the endpoint and capabilities
Put site-specific values in the platform environment file rather than changing the shipped defaults. This example uses a provider that supports bucket policies, SSE-S3, and versioning:
OBJECT_STORAGE_PROVIDER=external-s3
OBJECT_STORAGE_SCHEME=https
OBJECT_STORAGE_HOST=s3.internal.example.com
OBJECT_STORAGE_PORT=443
OBJECT_STORAGE_REGION=eu-west-1
OBJECT_STORAGE_PATH_STYLE=true
OBJECT_STORAGE_PUBLIC_HOST=s3.example.com
# Leave empty when the endpoint chains to a public CA already trusted by the platform.
OBJECT_STORAGE_CA_PEM="<PEM-encoded private CA certificate>"
OBJECT_STORAGE_CAPABILITIES_SSE=true
OBJECT_STORAGE_CAPABILITIES_PRESIGN_SIGNS_CONTENT_TYPE=false
OBJECT_STORAGE_CAPABILITIES_CROSS_ACCOUNT=policy
OBJECT_STORAGE_CAPABILITIES_STS_FEDERATION=false
OBJECT_STORAGE_CAPABILITIES_STS=false
OBJECT_STORAGE_CAPABILITIES_VERSIONING=true
OBJECT_STORAGE_CAPABILITIES_MANAGE_BUCKET_CORS=falseSet the capability values to match the service that was tested; they are not preference flags.
| Capability | Meaning for an external provider |
|---|---|
sse | Enables SSE-S3 requests only when the provider supports them |
presign_signs_content_type | Records whether the provider validates Content-Type on presigned uploads |
cross_account | Selects policy, posix, or none for shared-bucket access |
sts_federation | Must be false; Keycloak-to-RGW federation is specific to Rook-Ceph |
sts | Keep false unless the provider is verified to intersect an assumed-role session policy with the account policy |
versioning | Enables behavior that depends on bucket versioning only when the provider supports it |
manage_bucket_cors | Allows BullSequana AI to replace the bucket's complete CORS document |
Use OBJECT_STORAGE_PATH_STYLE=true when the provider expects https://host/bucket/key. Use false only when wildcard DNS and certificates support virtual-hosted requests such as https://bucket.host/key.
4. Hand off storage credentials
The storage operator provides one site bundle through the approved secret-handoff process. Include:
- each provider-issued access-key and secret-key pair
- the account-to-bucket mapping and allowed operations
- the MLflow account, which writes model artifacts to the shared
mlflowbucket - the tenant provisioner account limited to
*-databuckets, when private tenant MLflow is used - any additional policy or filesystem-group assignments used for cross-account access
The BullSequana AI deployment operator stores this bundle through the site's secret-management process and maps it into the protected platform environment. The platform currently uses separate configuration slots because it renders a Kubernetes Secret for each component. It does not require every slot to contain a different provider account.
Components that are disabled need no credential mapping. For the exact deployment variables, see Credential mapping reference.
5. Configure browser CORS
Presigned file downloads originate from the portal in the user's browser. Each affected external bucket must allow GET and HEAD from the portal origin.
A provider-neutral policy has this shape:
{
"CORSRules": [
{
"AllowedOrigins": ["https://portal.example.com"],
"AllowedMethods": ["GET", "HEAD"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "Content-Length", "Content-Type", "Content-Disposition", "Content-Range", "Accept-Ranges"],
"MaxAgeSeconds": 3600
}
]
}Prefer OBJECT_STORAGE_CAPABILITIES_MANAGE_BUCKET_CORS=false and let the storage operator own this policy. Set it to true only when the storage owner explicitly delegates the complete CORS document to BullSequana AI. PutBucketCors replaces the existing document rather than merging a rule into it. The setting covers default and default-artifacts only.
Buckets created at runtime
The platform creates these buckets after deployment, so neither a policy set before deployment nor manage_bucket_cors reaches them:
| Bucket | CORS |
|---|---|
<tenantId>-data | The platform sets no policy. Apply the policy above after the tenant operator creates the bucket, and repeat this for every new tenant. Do not create the bucket in advance: the operator refuses a bucket it did not create. |
user-*, space-* | The Kubeflow profile bucket provisioner replaces the policy with GET and HEAD for the Lakekeeper UI origin. If the provider refuses the call, the provisioner logs a warning and continues. |
kfp-* | The platform sets no policy, and none is needed. The KFP UI reads these artifacts on the server, not in the browser. |
Check which rule the provider applies to a new bucket by default. A default rule can allow every origin, which still requires a valid signature but is broader than the platform needs. On a <tenantId>-data bucket, a default rule that does not expose Accept-Ranges also makes the portal download a whole PDF to preview it.
6. Disable unused Ceph components
When the site does not use Rook-Ceph for object, block, or shared-file storage, disable its components:
ROOK_CEPH_COMPONENT_ENABLED=false
ROOK_CEPH_CLUSTER_COMPONENT_ENABLED=false
CEPH_CSI_DRIVERS_COMPONENT_ENABLED=falseDo not disable the CSI drivers until replacement RWO and RWX storage classes are configured and tested. Selecting external-s3 alone does not create those classes.
7. Validate and render
Before rendering, test each writable account against every bucket it is expected to use. When the provider supplies one shared account, repeat the bucket checks with that account:
export AWS_ACCESS_KEY_ID=<component-access-key>
export AWS_SECRET_ACCESS_KEY=<component-secret-key>
export AWS_DEFAULT_REGION=eu-west-1
export S3_ENDPOINT=https://s3.internal.example.com:443
aws --endpoint-url "$S3_ENDPOINT" s3api head-bucket --bucket <component-bucket>
aws --endpoint-url "$S3_ENDPOINT" s3 cp ./s3-smoke-test.txt "s3://<component-bucket>/s3-smoke-test.txt"
aws --endpoint-url "$S3_ENDPOINT" s3 cp "s3://<component-bucket>/s3-smoke-test.txt" ./s3-smoke-test.downloaded.txt
aws --endpoint-url "$S3_ENDPOINT" s3 rm "s3://<component-bucket>/s3-smoke-test.txt"For a read-only account, replace the write and delete operations with head-object and a download of a known test object. Also test the MLflow bucket with the MLflow writer and the KServe reader credentials.
Render the complete desired state:
bsqai apply all -rRendering fails closed when the provider, host, scheme, cross-account mode, STS federation setting, or required component credentials are inconsistent. This includes the tenant operator's BSQAI_TENANT_OPERATOR_S3_* pair unless the site manages the provisioner Secret itself. Treat those errors as configuration failures; do not replace missing external credentials with generated placeholder values.
Review the generated manifests before pushing them. Confirm that:
- endpoints include the intended explicit port
- the public hostname is correct for presigned URLs
- private CA material is mounted where required
- external credentials are rendered as Sealed Secrets
- Rook bucket and user resources are absent for external-S3 consumers
- disabled Ceph applications are absent from the generated app-of-apps output
8. Verify after deployment
Run an end-to-end check for every enabled storage path:
- Upload and download a file through the portal, in the Global organization and in a tenant, then verify the browser reports no CORS error.
- Process a file for RAG and confirm that Milvus search returns it.
- Create and verify a CloudNativePG backup in
cnpg-backups. - Confirm Loki logs, Tempo traces and Mimir metrics continue to appear in Grafana.
- Import or deploy a model and verify the MLflow writer and KServe reader paths.
- Run a Kubeflow pipeline and confirm its artifacts, if Kubeflow is enabled.
- Verify managed-ingestion checkpoint backups and an Argo Events artifact flow when enabled. Separately test any data ingestion connection that uses this object store as a source or destination, then verify its ingestion result.
- If the agent sandbox is enabled, leave a sandbox session of a tenant idle until the reaper stops it, confirm that its workspace objects appear under
sandbox/<tenantId>/<chat-id>/in the tenant's<tenantId>-databucket, and confirm that the next message in the chat restores the files.
Inspect provider audit logs for denied requests, unexpected bucket access, TLS failures, or requests sent to the public endpoint when the private endpoint was expected.
Migration and rollback
Changing OBJECT_STORAGE_PROVIDER changes where the platform looks for objects; it does not copy data from Rook-Ceph or rewrite existing object references.
Before switching an existing deployment:
- stop or quiesce writers
- copy every required bucket, object version, and metadata record
- validate object counts, sizes, checksums, retention, and restore procedures
- update the endpoint, capabilities, credentials, and Ceph component settings together
- render and review the complete desired state
- keep the previous store read-only until application-level validation succeeds
Rollback requires the previous endpoints, credentials, manifests, and object contents to remain available. Do not delete or reuse the old buckets as part of the initial cutover.
Tempo verifies the endpoint certificate, like the other storage consumers. Earlier releases skipped this check, so an existing site can have a problem that only appears at upgrade. Before you sync the upgrade, check that:
- a private or self-signed certificate is trusted through
OBJECT_STORAGE_CA_PEM, or through the Secret and key named byOBJECT_STORAGE_CA_SECRETandOBJECT_STORAGE_CA_KEY, which must exist in the Tempo namespace - the certificate has a subject alternative name that matches
OBJECT_STORAGE_HOST
Otherwise Tempo stops at startup with an x509 error.
The managed-ingestion backup identity is separate from the credentials of data ingestion connections. Source and destination credentials belong to their connections and need the appropriate native bucket permissions. See managed ingestion.
Credential mapping reference
This mapping is performed by the BullSequana AI deployment operator. Set the pair for every enabled S3 consumer in the protected platform environment. A dedicated provider account is recommended for each consumer. When the provider supplies one shared account, map the same pair into each required slot.
GRAFANA_LOKI_S3_ACCESS_KEY=<access-key>
GRAFANA_LOKI_S3_SECRET_KEY=<secret-key>
TEMPO_S3_ACCESS_KEY=<access-key>
TEMPO_S3_SECRET_KEY=<secret-key>
MIMIR_S3_ACCESS_KEY=<access-key>
MIMIR_S3_SECRET_KEY=<secret-key>
MILVUS_S3_ACCESS_KEY=<access-key>
MILVUS_S3_SECRET_KEY=<secret-key>
MLFLOW_S3_ACCESS_KEY=<access-key>
MLFLOW_S3_SECRET_KEY=<secret-key>
KSERVE_RESOURCES_S3_ACCESS_KEY=<access-key>
KSERVE_RESOURCES_S3_SECRET_KEY=<secret-key>
MELTANO_S3_ACCESS_KEY=<access-key>
MELTANO_S3_SECRET_KEY=<secret-key>
ARGO_EVENTS_S3_ACCESS_KEY=<access-key>
ARGO_EVENTS_S3_SECRET_KEY=<secret-key>
KUBEFLOW_S3_ACCESS_KEY=<access-key>
KUBEFLOW_S3_SECRET_KEY=<secret-key>
LAKEKEEPER_S3_ACCESS_KEY=<access-key>
LAKEKEEPER_S3_SECRET_KEY=<secret-key>
LLM_BACKEND_S3_ACCESS_KEY=<access-key>
LLM_BACKEND_S3_SECRET_KEY=<secret-key>
CNPG_BACKUPS_ACCESS_KEY_ID=<access-key>
CNPG_BACKUPS_ACCESS_SECRET_KEY=<secret-key>
BSQAI_TENANT_OPERATOR_S3_ACCESS_KEY=<tenant-provisioner-access-key>
BSQAI_TENANT_OPERATOR_S3_SECRET_KEY=<tenant-provisioner-secret-key>The tenant operator pair is the provisioner account limited to *-data buckets. The platform renders it as the tenant-storage-provisioner Secret. A site that manages that Secret outside the platform sets BSQAI_TENANT_OPERATOR_STORAGE_PROVISIONER_SECRET_NAME to its name instead; the pair is then not required.
The Model Installer has two additional configuration fields for the mlflow bucket. Set them to the MLflow account's pair, the same values as MLFLOW_S3_*.
LLM_BACKEND_MLFLOW_S3_ACCESS_KEY_ID=<mlflow-access-key>
LLM_BACKEND_MLFLOW_S3_SECRET_ACCESS_KEY=<mlflow-secret-key>Do not commit these values. The platform seals the rendered Kubernetes secrets, but the source credentials remain in the site's secret-management process.
Troubleshooting
AccessDeniedduring rendering or startup — verify the mapped credential and its bucket policy for the affected component.- TLS verification failure — supply the private CA through
OBJECT_STORAGE_CA_PEMor fix the endpoint certificate chain. A certificate with only a common name fails withx509: certificate relies on legacy Common Name field; reissue it with a subject alternative name. - Requests use the wrong port — set
OBJECT_STORAGE_PORTexplicitly and verify the rendered endpoint. - Browser download fails while backend access works — verify
OBJECT_STORAGE_PUBLIC_HOST, its certificate, and the bucket CORS policy, including<tenantId>-datafor a tenant. - Presigned upload fails only on one provider — verify
OBJECT_STORAGE_CAPABILITIES_PRESIGN_SIGNS_CONTENT_TYPEagainst that provider's signing behavior. - MLflow import succeeds but model loading fails — test the
mlflowbucket with the KServe credential and review cross-account enforcement. - A tenant with private MLflow reports
StorageProviderConfigurationMissing— the tenant operator has no provisioner Secret; set theBSQAI_TENANT_OPERATOR_S3_*pair or the externally managed Secret name and render again. - A tenant with private MLflow reports
InvalidStorageBinding— this can mean that the tenant's S3 endpoint is not HTTPS or embeds credentials, that the tenant has no storage binding, or that its MLflow S3 credential Secret has the wrong name or owner. Check the endpoint in the tenant'sstatus.storage.endpoint. SettingOBJECT_STORAGE_SCHEME=httpsdoes not repair a tenant that already has storage on an HTTP endpoint; it causesStorageBindingImmutable. - A tenant reports
StorageBindingImmutable— the configuration no longer matches the binding recorded when the tenant's storage was provisioned. After a change to the endpoint, region, or provisioner Secret, every tenant with storage reports this. Restore the previous values and render again. Moving a provisioned tenant to another endpoint is an explicit migration; the platform does not move the binding or the data. Do not delete the<tenantId>-storage-bindingConfigMap to force the change. - Private MLflow cannot reach an endpoint on a port other than 443 — its egress policy has a separate port setting. Set
BSQAI_TENANT_OPERATOR_MLFLOW_STORAGE_PORTto the endpoint port and render again. See External S3. - PostgreSQL backups fail — verify the
cnpg-backupsbucket, the twoCNPG_BACKUPS_*credentials, retention support, and CA trust.