Configure private tenant MLflow

Configure, verify, and operate private tenant MLflow with protected backend access and retained artifact storage.

Agentic Friendly

Use this playbook as a platform operator to configure private tenant MLflow and verify its lifecycle. The result is a tenant service reachable over verified HTTPS from the configured BSQAI API pods. Human authentication and resource authorization belong to the BSQAI API. When the service is ready, the BSQAI API and Studio serve the tenant organization's MLflow content from it.

Prerequisites

  • Installation configuration access and an explicit Kubernetes context for the intended cluster.
  • Matching runtime operator, CRDs, RBAC, platform configuration, and an approved MLflow image with native authentication and PostgreSQL and S3 dependencies.
  • Compatible BSQAI API and Portal versions with tenant service management and tenant-aware MLflow consumption, plus platform administrator access to manage service state and grant tenant service-management permission.
  • A cluster-wide grant that lets the BSQAI API service account get Secrets named mlflow-runtime-credentials. The platform manifests render it as the ClusterRole and ClusterRoleBinding llm-backend-mlflow-tenant-credentials, limited by resource name, with no list or watch. Without it, every customer organization with MLflow enabled receives 503 mlflow_service_unavailable.
  • A ca-bundle Secret, mounted in the BSQAI API pods, that trusts the issuer of the tenant MLflow certificates.
  • CloudNativePG with Database and DatabaseRole reconciliation, verified PostgreSQL TLS, and the existing tenant identity control plane.
  • Object storage with verified HTTPS access to the selected S3 endpoint: Rook-Ceph RGW for automatically scoped artifact credentials, or an external S3 provider with a provisioner identity limited to tenant buckets (see External S3).
  • cert-manager with an issuer that publishes a CA bundle and issues certificates for internal Service names.
  • An enforcing network-policy engine, cluster DNS, and permission to inspect the tenant namespace.

Private MLflow runs in the tenant namespace, tenant-<tenantId>, next to the tenant's model-serving and other runtime workloads, such as Managed MCP Servers. Do not grant tenant users Kubernetes access to that namespace, and do not distribute backend machine credentials to users or notebook pods.

1. Configure installation defaults

Use installation-level configuration for provider endpoints, images, trust, and network selectors. Do not add per-tenant GitOps entries for automatic provisioning.

Pin a matching published operator chart and image with private MLflow support. Apply the CRDs, RBAC, and installation configuration with that operator, then verify operator readiness. Apply the BSQAI API credential-read grant before you roll out an API version that consumes tenant MLflow. Deploy the compatible BSQAI API consumer after this control plane is ready, including its database migrations and lifecycle workers. Deploy the matching Portal after the API exposes the tenant service-management endpoints and GET /v1/mlflow/capabilities. Changing chart values alone does not upgrade an older controller.

The operator chart consumes these fields. The platform renderer derives provider and backend settings from the shared object-storage configuration and backend Helm release.

Operator chart fieldConfiguration to verify
storage.defaultEndpoint, storage.defaultRegionThe selected S3 endpoint and signing region
storage.credentialProviderrook-ceph for automatic scoped credentials, or external-s3
storage.provisionerSecretNameExternal S3 only: the operator-namespace Secret with the provisioner's accessKeyId and secretAccessKey. The platform renders tenant-storage-provisioner from bsqai-tenant-operator s3.access_key and s3.secret_key unless a Secret managed outside the platform is named here
storage.rookNamespace, storage.rookStoreThe Rook namespace and object store that own the provider identities
mlflow.imageThe approved MLflow image with native authentication support
mlflow.backendNamespace, mlflow.backendPodSelectorThe actual API namespace and a nonempty pod selector that match the same peer
mlflow.tlsIssuerName, mlflow.tlsIssuerKindThe internal certificate issuer, normally a ClusterIssuer
cnpg.defaultClusterName, cnpg.clusterNamespaceThe shared PostgreSQL cluster and namespace
mlflow.postgresCASecretNameThe CA Secret in the CNPG namespace. Empty selects <cluster>-ca with key ca.crt
mlflow.s3CASecretName, mlflow.s3CASecretKeyThe S3 CA bundle in the operator namespace. An empty Secret name uses system roots
mlflow.storageNamespace, mlflow.storagePodSelector, mlflow.storagePortThe in-cluster storage destination and actual destination pod port
mlflow.storageExternalCIDRsExplicit storage destinations outside the cluster when required
mlflow.dnsTargetsThe actual DNS namespace or NodeLocal DNS IP and port
mlflow.imagePullSecretNameThe registry Secret supplied to tenant namespaces through the installation's Secret reflection mechanism

Check the rendered backend labels against the running API pods. Both namespace and pod selector restrict ingress. An unrelated allow policy can broaden access because Kubernetes NetworkPolicies are additive.

For S3 egress through a gateway, verify the destination pod and port after network address translation. Standard NetworkPolicy cannot restrict a destination by DNS hostname. Validate the actual path with the installation's network-policy engine.

PostgreSQL connections use sslmode=verify-full. The CA must trust <cluster>-rw.<namespace>.svc.cluster.local. Rotate PostgreSQL and S3 roots with overlapping bundles before removing the old roots.

No public MLflow DNS name, Gateway route, browser OIDC client, or mlflow.authorizationReady setting is required. Private tenant MLflow does not load the browser OIDC plugin.

External S3

External S3 has no API to create a scoped identity per tenant. With storage.credentialProvider: external-s3, the operator creates and owns each tenant's <tenantId>-data bucket with the provisioner identity and gives that identity to the tenant's private MLflow, which writes under mlflow/. It never adopts a bucket it did not create. Disabling MLflow removes the credential from the tenant namespace and keeps the bucket and its data.

Tenants never receive this credential and cannot reach the private service; only the BSQAI API can. The provisioner identity can still reach every tenant bucket, so a compromised MLflow process could read other tenants' artifacts. Limit the identity to tenant buckets, for example a policy on *-data and *-data/*, and give it no other access. An administrator-supplied scoped Secret named <tenantId>-mlflow-s3 takes precedence for a tenant.

For an S3 service running inside the cluster, such as MinIO, set bsqai-tenant-operator mlflow.storage_namespace, mlflow.storage_pod_label (key=value) and mlflow.storage_port to the service's namespace, a pod label and the pod port. For an endpoint outside the cluster, set mlflow.storage_external_cidrs. The platform render fails on external S3 until s3.access_key and s3.secret_key are set, unless storage.provisioner_secret_name names a Secret the site manages itself. Without a provisioner Secret, enabling MLflow reports StorageProviderConfigurationMissing.

2. Check storage ownership before activation

The Rook-Ceph checks below do not apply to external S3; see External S3.

On Rook-Ceph, for fresh implicit MLflow storage, let the operator create the lifecycle-owned bucket owner, artifact bucket, and separate workload identity. The workload identity receives artifact access under mlflow/ and cannot create buckets.

For an existing ordinary tenant bucket, verify that its journal and target reservation match the current tenant UID and lifecycle. The recorded endpoint and region must match the configured Ceph provider. The operator can then attach a generated MLflow identity without changing the bucket owner or creating another bucket.

Keep the bucket-owner identity restricted to trusted administration. General tenant credentials must belong to a different provider user. Different keys for the same owner do not satisfy this condition. The operator verifies provider identity and bucket ownership before publishing the artifact binding.

For both newly managed buckets and existing shared tenant buckets, the operator preserves unrelated policy statements and reads the applied policy back before publishing credentials. Explicit deny rules fence the reserved prefix, bucket administration, and excess MLflow permissions even when another statement grants broad access. General users cannot list the bucket root or ancestors of mlflow/. Bucket-wide multipart-upload listing is denied to non-owners. Ordinary operations on permitted prefixes remain available.

If a journal already records externally managed MLflow credentials, keep the instance disabled until an explicit migration verifies ownership and revokes old workload keys. Do not delete its journal or ObjectBucketClaim to force reprovisioning. Changing the provider setting does not transfer ownership or move data.

External S3 uses the provisioner identity instead of scoped identities; see External S3. Existing administrator-provisioned bindings have a separate credential and provider-revocation procedure.

3. Enable the tenant capability

In the Portal, open Administration → Tenants, select the tenant, and open Services. A platform administrator selects Enable MLflow. A tenant administrator may do so when a platform administrator has granted tenant.service.manage for that tenant. For a new tenant, a platform administrator selects MLflow in the creation wizard's Services step. Grant management permission separately after the tenant Organization is provisioned. See Manage private MLflow for permission and recovery behavior.

The API submits the activation request to the lifecycle workflow; the operator receives spec.mlflow.enabled: true. Omission leaves MLflow disabled. Use a tenant slug of at most 50 characters.

The BSQAI API sets spec.mlflow.reclaimPolicy: Retain, so disabling MLflow keeps its data, and sets Delete just before it deletes the tenant, so the MLflow metadata database goes with the tenant. A PlatformTenant managed outside the BSQAI API keeps Retain unless its approved deletion policy requires data removal. MLflow implicitly requires storage even when the general storage switch is off. Its metadata database is separate from the optional tenant application database.

In the tenant namespace, the operator creates the private Service, certificate, a network policy that selects only MLflow's pods, metadata persistence, and machine identity. It does not create an end-user login or move shared MLflow experiments and artifacts.

Export the existing PlatformTenant name and inspect its current observations:

export TENANT_RESOURCE="<PlatformTenant-name>"
kubectl get platformtenant "$TENANT_RESOURCE" -o json | jq '{
  generation: .metadata.generation,
  observedGeneration: .status.observedGeneration,
  mlflow: .status.mlflow,
  conditions: [.status.conditions[]? | select(.type == "MLflowReady" or .type == "StorageReady")]
}'

Accept the runtime binding only when MLflowReady=True, the condition's observedGeneration equals metadata.generation, and status.mlflow.observedGeneration matches that generation. A stale success condition does not describe the current request.

Check that status.mlflow contains accessMode: backend-only, an internal HTTPS URL, and credential and CA references in the tenant namespace. The URL has the form https://mlflow.tenant-<tenantId>.svc.cluster.local:5000. Read the published binding instead of constructing a consumer URL.

The credential Secret mlflow-runtime-credentials contains only username and password. The CA reference supplies public trust material. References are operational metadata, not content grants. Keep Secret delivery restricted to authorized backend workloads.

4. Verify the access boundary

Use an approved test tenant and an authorized diagnostic workload for these checks. Keep credentials out of command output, shell history, and logs.

  1. From a pod matching the configured backend namespace and selector, verify the published hostname and CA chain. Do not bypass TLS verification.
  2. Use the tenant's backend machine identity to create and read a test MLflow run and artifact. Compare the downloaded artifact with the uploaded bytes.
  3. Confirm that anonymous requests, browser sessions, bearer tokens, and another tenant's machine credentials cannot authenticate.
  4. Confirm that Portal pods, tenant workload pods, including model-serving pods in the same tenant namespace, and sibling pods outside the backend selector cannot reach the private service, and that tenant workload service accounts cannot read its Secrets mlflow-credentials, mlflow-runtime-credentials, mlflow-s3-credentials and mlflow-server-tls (kubectl auth can-i get secret/<name> -n tenant-<tenantId> --as system:serviceaccount:tenant-<tenantId>:<account>).
  5. On Rook-Ceph, confirm that artifact credentials cannot access another bucket or a key outside the reserved prefix, and that general tenant credentials cannot access mlflow/. On external S3, confirm that the artifact credential is denied outside *-data buckets; sibling tenant buckets remain reachable by design.
  6. Check that no public route serves the private tenant instance.
  7. As a tenant member with the AI Builder role, call GET /v1/mlflow/capabilities with the tenant organization selected. Expect available: true, scope: "tenant", and service_state: "ready".
  8. In that organization, create a test experiment through Studio or the BSQAI API. Confirm that the Global organization and another tenant organization cannot see it, and that a request for its run ID from another organization returns 404.

For step 7, send the request with a user access token and the tenant organization's UUID:

export BSQAI_BASE_URL="https://api.<platform-domain>/v1"
export BSQAI_TOKEN="<jwt-access-token>"
export BSQAI_TENANT_ID="<keycloak-organization-uuid>"
curl -sS "$BSQAI_BASE_URL/mlflow/capabilities" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "X-Tenant-Id: $BSQAI_TENANT_ID"

The BSQAI API resolves the MLflow of the active organization on every request. The Global organization uses the installation's MLflow. A tenant organization uses its private instance only while its service state is ready. In every other state, /v1/mlflow routes return 503 mlflow_not_enabled and send no request to any MLflow server. The API never falls back to the Global MLflow or to another organization's instance.

For a ready tenant, 503 mlflow_service_unavailable comes from a platform fault. Check the credential-read grant, the mlflow-runtime-credentials Secret, and CA trust. A tenant organization deploys and imports models from its private MLflow, with the model files under s3://<slug>-data/mlflow/; see Organizations and namespaces. Copying a model from other S3 storage with download_s3_model, deleting models with delete_model, MLflow workspaces, and MLflow webhooks stay in the Global organization. See MLflow and the active organization for the error codes.

5. Verify disable and re-enable

In the tenant's Services tab, select Disable MLflow. Revoking tenant service-management permission leaves enabled services running. Wait for the current command and operator observation to converge. While the service is disabling or disabled, GET /v1/mlflow/capabilities reports available: false with that service_state, and /v1/mlflow routes return 503 mlflow_not_enabled. The Global organization and other tenant organizations are unaffected.

Verify that the MLflow workload has stopped, the runtime binding and workload credential mirrors are absent, the tenant namespace and its other workloads are unaffected, and database login is disabled. On Rook-Ceph, also verify that the managed artifact identity has lost provider access; check a previously copied managed key as well as the Kubernetes Secrets. Removing a Secret alone does not prove provider revocation. On external S3, disabling removes the credential copy and the workload but does not revoke the provisioner key; rotate it if a copy may have been captured.

For an existing shared tenant bucket, verify that ordinary storage remains usable and the reserved-prefix fence remains in place. Disablement retains the bucket, metadata, ownership journal, and recorded artifact location.

Select Enable MLflow for the same tenant lifecycle. During provisioning, routes still return mlflow_not_enabled. Wait for current-generation readiness, then read the earlier run through the BSQAI API and compare its artifact bytes again. Re-enable restores the recorded location. It does not adopt data from another tenant with the same slug.

Tenant deletion follows the storage and MLflow reclaim policies; for tenants deleted through the BSQAI API, both are Delete. Either capability's retained policy preserves a shared bucket. Delete can remove data and must follow the approved tenant deletion process. Keep the supporting controller and ownership records until cleanup completes.

Remove MLflow data a deleted tenant left

A tenant deleted with Retain leaves its MLflow metadata in the CloudNativePG namespace: the <slug>.mlflow Database and DatabaseRole resources and Secret, and in the operator namespace an app-target-* ConfigMap whose target is postgres:<cnpg namespace>/<cluster>/<slug>_mlflow. Tenants deleted before the BSQAI API asked for Delete left them this way. Once no PlatformTenant has that tenantId:

  1. Set spec.databaseReclaimPolicy: delete on the Database and delete it. CloudNativePG drops the database.
  2. Set spec.databaseRoleReclaimPolicy: delete on the DatabaseRole and delete it. CloudNativePG drops the role.
  3. Delete the <slug>.mlflow Secret and the app-target-* ConfigMap.

Deleting either resource without its reclaim policy set to delete leaves the PostgreSQL database or role in the cluster.

6. Upgrade and roll back

Back up provider data and ownership journals before every platform upgrade. Upgrade the runtime operator, the BSQAI API and the Portal in the order of 1. Configure installation defaults.

Preserve the Global MLflow instance and its clients. The operator never moves data between the Global MLflow and a private instance, and a tenant organization reads only its own private MLflow. Plan any data move separately.

Before an operator rollback, disable private MLflow and withdraw dependent API traffic. Roll back the dependent API consumer first if required. Keep a compatible operator while private resources or cleanup finalizers remain. An older image does not reverse tracking or authentication schema upgrades, so restore a tested compatible release or follow a separate database recovery plan.

If ownership validation fails, restore the original verified journal and credential records. Do not reconstruct ownership from matching names, bind retained data to a new UID, or remove cleanup finalizers.

On this page