BSQAI Tenant Operator
Kubernetes operator that reconciles each tenant's namespace, quotas, identity, storage, databases, model serving, and telemetry resources.
Component Category
tenant lifecycle
Component Description
BSQAI Tenant Operator reconciles a cluster-scoped PlatformTenant resource into the Kubernetes and provider resources required by one tenant. It also reconciles independent application buckets and databases through PlatformTenantResource. Status reports readiness, observed generations, provider bindings, and lifecycle conditions.
Why It Is Used
In BullSequana AI, the operator makes tenant provisioning idempotent and observable. For each tenant it reconciles:
- the tenant namespace with its
ResourceQuota,LimitRange, and default-deny NetworkPolicy; - a Keycloak Organization and, when configured, the tenant's external OIDC or SAML identity provider;
- optional S3 buckets and an optional tenant-bound S3 role for web-identity access;
- an optional CloudNativePG database and role;
- optional tenant Milvus and private MLflow resources;
- model serving resources when inference is enabled: KAI Scheduler GPU queues, the deployment allowance, the model storage credential and identity, and the BSQAI API's binding in the serving namespace;
- the namespace preparation for the organization's Managed MCP Servers;
- a Grafana Organization for the tenant's telemetry.
Deletion runs those reconcilers in reverse and remains protected by a finalizer until cleanup succeeds.
The tenant namespace is always tenant-<tenantId>, where spec.tenantId is the tenant's slug, and PlatformTenant rejects any other spec.namespace. The operator records the tenant lifecycle on the namespace when it creates it, in the bsqai.io/lifecycle-id annotation, and uses an existing namespace only when it carries the tenant's labels and lifecycle. A same-named namespace without them is reported as NamespaceOwnershipConflict and is never changed or deleted, also when the tenant is deleted. A namespace that carries the tenant's labels but no lifecycle annotation stays owned and receives the annotation on its next reconcile.
Learn More
Lifecycle flow
Phases and conditions
status.phase summarizes the tenant:
| Phase | Meaning |
|---|---|
Pending | The operator accepted the tenant but has not reconciled it yet. |
Provisioning | Resources are being created or updated, or a condition is Unknown while it waits for something. |
Ready | Every gating condition is True for the current generation. |
Degraded | A gating condition is False, after a failure or when a capability that was ready breaks. |
Deleting | The tenant is being deleted. The finalizer stays until cleanup succeeds. |
The operator reconciles the capabilities in a fixed order. When one fails, the later ones report reason DependencyNotReady until it recovers: Unknown while they have not yet been reported for the current generation, False once they have. A capability that the tenant does not request reports True, so it does not hold back Ready.
| Condition | Reports | Gates Ready |
|---|---|---|
NamespaceReady | The namespace, its quota and limit range, its NetworkPolicies and, with inference, the serving resources | Yes |
IdentityReady | The Keycloak Organization and its groups | Yes |
StorageReady | The tenant buckets and, with inference, the model storage credential | Yes |
DatabaseReady | The tenant application database | Yes |
MilvusReady | The tenant Milvus database, user, and role | Yes |
MLflowReady | Private MLflow | Yes |
FineTuningReady | Reserved for a capability that tenants do not request in this release, so it reports True | Yes |
ObservabilityReady | The tenant Grafana Organization; reported only while the operator manages Grafana | Yes |
IdentityProviderReady | The tenant's external identity provider broker. LocalAuthentication means external login is off. A broker failure also fails IdentityReady. | No |
McpServersReady | Whether the BSQAI API may run Managed MCP Servers in the namespace | No |
ObjectStorageAccessReady | The tenant-bound S3 role; reported only while the role is configured | No |
ModelStorageScoped | Whether the model storage credential reaches only this tenant's artifacts; reported only with inference | No |
Platform administrators read the phase and conditions in the Portal under Administration > Tenants > the tenant > Lifecycle health. See Inspect tenant status.
Deployment notes
The operator runs in bsqai-system without a public endpoint. It uses a Keycloak service account, S3 provider bindings, and the CloudNativePG Database and DatabaseRole APIs. Private MLflow also uses cert-manager, and Rook-Ceph credential provisioning or, on external S3, the storage provisioner identity. Install matching operator, CRD, RBAC, and platform configuration versions.
The immutable tenant slug names Kubernetes resources and the Keycloak Organization alias. The Organization UUID reported in status is the canonical runtime tenant identifier.
The application database check connects to the shared CloudNativePG cluster as the tenant's own role over verified TLS. It needs the cluster's <cluster>-ca Secret in the CNPG namespace and operator egress to the cluster on port 5432.
While [network_policy] enabled is on, the operator applies the NetworkPolicy tenant-default-deny in every tenant namespace. A tenant pod that no other policy covers then accepts and sends traffic only within its namespace. MCP server pods and model-serving pods keep their own policies while those are in place, and private MLflow always does. Model-serving pods keep open outbound traffic, because they install packages, fetch weights, and may use the tenant's external S3 storage. BSQAI_TENANT_OPERATOR_NETWORK_POLICY_TENANT_DEFAULT_DENY=false turns off this policy alone, and the operator deletes it. Like every policy here, it takes effect only with an enforcing network-policy engine. See Network policies.
Optional tenant capabilities
PlatformTenant can provision tenant-scoped Milvus and private MLflow resources in addition to identity, storage, and the optional application database. Omitted Milvus/MLflow capabilities are disabled. They default to retaining data; disabling a capability withdraws access rather than deleting its data. The BSQAI API asks for Delete on MLflow when it deletes a tenant.
Milvus provisioning creates a database, user, and private role on an approved provider. It does not create document collections or grant access to Files. Configure authenticated HTTPS access and CA trust, and remove shared public-role grants before enabling tenant isolation. The operator fails closed when the provider's shared grants invalidate that boundary.
Private MLflow has separate tracking and authentication persistence, a reserved artifact prefix, and a backend machine identity. The operator runs it in the tenant namespace, tenant-<tenantId>, with a private HTTPS ClusterIP Service and a NetworkPolicy that selects only MLflow's pods and permits ingress only from pods matching both the configured backend namespace and pod selector. Tenant workloads in the same namespace and human browser sessions have no direct access, and tenant workload service accounts cannot read MLflow's Secrets. The BSQAI API owns human and resource authorization.
MLflow requires artifact storage even when the general storage switch is off. On Rook-Ceph, the operator provisions scoped credentials. It can create a dedicated bucket or attach a scoped identity to a verified existing tenant bucket without changing that bucket's owner. External S3 has no scoped-identity API, so the operator creates and owns the tenant's <tenantId>-data bucket with the storage provisioner identity and gives that identity to private MLflow, unless an approved scoped <tenantId>-mlflow-s3 Secret exists. Tenants never receive it and only the BSQAI API reaches the service, but the identity can reach every tenant bucket. See External S3.
This capability is separate from the installation's Global MLflow. Enabling it does not move existing experiments or model artifacts. When the capability is ready, the BSQAI API serves the tenant organization's experiments and model repository from it, in Studio and under /v1/mlflow. In any other state, the API reports that the service is not enabled and never falls back to the Global MLflow.
Tenant application database
When PostgreSQL is selected for a tenant, the operator creates a database named after the tenant slug on the shared cnpg-cluster. The role <tenantId>_owner owns it, and only the operator uses that role: its password stays next to the cluster.
Workloads in tenant-<tenantId> connect to cnpg-cluster-rw.cnpg.svc:5432 over TLS as the role <tenantId>_app, using the username and password keys of the tenant-db-credentials Secret in that namespace. That role can connect, use temporary tables, and create and use objects in the public schema. It does not own the database, so it cannot change it, create roles, or give another role access.
Only pods labelled bsqai.io/tenant-database: "true" reach the cluster. That limit needs an enforcing NetworkPolicy engine; without one, every pod can reach the cluster and only PostgreSQL privileges apply. Each app role holds at most 20 connections by default. The operator's cnpg.tenantConnectionLimit sets the cap, and 0 or less means unlimited.
The database is isolated before it is reported ready:
- The operator creates it with connections disabled.
- As the owner, it revokes the
PUBLICprivileges on the database and itspublicschema, plus the defaultEXECUTEon future functions. Only then does it allow connections. - On every check, it revokes every other grant on the database, including access passed on with a grant option, and gives the app role exactly its privileges. The owner and the app role must then be the only roles with access. Neither may have elevated attributes or hold or grant a role membership, and the app role must not own the database.
- It ends every other session of the owner role, such as one opened with an earlier password.
DatabaseReadyturnsTrueonce no session remains from a role that may no longer connect.
Only then does a NetworkPolicy next to the cluster admit the tenant's labelled pods. The operator removes that policy as soon as a later check fails, when the database is disabled, and when the tenant is deleted.
This keeps other tenants' roles out of the database, and tenant roles out of the platform's databases, which are closed the same way. It assumes that the owner's credentials and the PostgreSQL superusers stay with the platform: superusers can reach every database. The NetworkPolicy is a second boundary only where an enforcing engine runs. On the shared cluster, tenants can still list database and role names, and they share its CPU, memory, and I/O.
Managed MCP Servers
The operator prepares every tenant namespace for the organization's Managed MCP Servers. It needs these operator values, which the platform sets:
| Value | Purpose |
|---|---|
mcpServers.backendClusterRoleName | The platform's mcp-servers-backend ClusterRole, bound in each tenant namespace. Unset disables the capability. |
mcpServers.backendServiceAccounts | The BSQAI API identities the role is bound to, each as namespace/name. |
mcpServers.callers | The only pods admitted to MCP server pods, each as a namespace and pod labels: the MCP Gateway, the BSQAI API and the MCP operator. |
mcpServers.imagePullSecretName | The pull Secret MCP server pods use for private images. Optional. |
mcpServers.egressCIDRs, mcpServers.egressPorts, mcpServers.dnsTargets | The only addresses, TCP ports, and DNS servers MCP server pods reach. The platform sets them from [network_policy] and MCP_LIFECYCLE_OPERATOR_NETWORK_POLICY_SERVER_EGRESS_PORTS. Empty CIDRs leave outbound traffic open. |
When the role is set but the service accounts, callers, or egress values are missing or invalid, the operator withdraws the binding and reports McpServersReady False with reason InvalidConfiguration, and the BSQAI API refuses to create Managed Servers in that tenant.
With those values in place, it creates two NetworkPolicies on MCP server pods: mcp-servers-isolation, which admits nobody, and mcp-servers-allow-callers, which adds the MCP Gateway, the BSQAI API and the MCP operator on any port. If either drifts, the other still keeps tenant workloads out. While mcpServers.egressCIDRs is set, a third policy, mcp-servers-egress, limits outbound traffic to the DNS targets and those addresses on the listed ports. It creates the mcp-server ServiceAccount the pods run as, without an API token. Then it binds the platform's mcp-servers-backend ClusterRole to the BSQAI API identities through a RoleBinding of the same name. When the tenant default-deny NetworkPolicy is enabled, it leaves MCP server pods to their own policies, and McpServersReady stays False with reason McpServersPolicyNotExclusive until the default-deny no longer selects them.
The condition McpServersReady reports the result. It never changes the tenant phase, so a failing unrelated capability does not block Connectors. The BSQAI API places servers in the namespace only while the condition is True. With the role unset, the operator removes the binding, the ServiceAccount, and the three policies, and reports the condition False with reason Disabled. A same-named object the operator did not create is reported as McpServersOwnershipConflict and never changed or deleted. Tenant deletion removes the binding first, after the BSQAI API has removed the organization's servers.
Private MLflow prerequisites
Configure the approved image, CNPG and S3 TLS trust, cert-manager issuer, backend selectors, and storage egress before enabling the capability. Cluster DNS and an enforcing network-policy engine are required. Private MLflow needs no public DNS name, Gateway route, browser OIDC client, or mlflow.authorizationReady flag.
The tenant MLflow tracking database (<tenantId>_mlflow) on the shared cluster is isolated like the tenant application database, with the MLflow role as its owner and only role with access. The operator creates it with connections disabled, revokes its PUBLIC privileges before connections open, and revokes any other grant on every check. While it is not isolated, MLflowReady is not True, and the operator stops MLflow and withdraws its binding. The condition is Unknown while the database is first set up, and False if a database that was ready loses its isolation.
The operator publishes status.mlflow with accessMode: backend-only, an internal HTTPS URL, and protected credential and CA references. A consumer accepts the binding only when MLflowReady=True and both the condition and binding observe the current resource generation. Secret references and infrastructure readiness do not grant content access.
Disabling MLflow stops its workload, withdraws its runtime binding and managed artifact access, and disables database login while retaining data. Re-enabling restores the recorded artifact location. Tracking and authentication schema upgrades are not reversed by selecting an older image. Use Configure private tenant MLflow for installation, verification, and upgrade checks.
Tenant object storage access
With the bsqai-tenant-operator component's [rgw_sts] mode set to enabled, the operator creates one Rook-Ceph RGW role for each tenant, named TenantS3Role-<tenantId> under the path /bsqai/tenant-operator/. The role grants access to the tenant's tenants/<tenantId>/ prefix of the lakekeeper bucket. It accepts only web-identity tokens from the platform realm whose tenant_id claim carries the tenant's Keycloak Organization UUID, so a token from one tenant cannot assume another tenant's role. Spark jobs use it through AssumeRoleWithWebIdentity. status.objectStorageAccess publishes the role's name and ARN. The roles of the Organizations listed in the Keycloak component's configuration, such as Global, come from a platform job instead, and their condition reports StaticRoleManaged.
The mode is empty by default: the operator then leaves RGW roles untouched and reports no ObjectStorageAccessReady condition. disabled revokes the roles the operator created. The condition never changes the tenant phase. A same-named role the operator did not create is reported as RoleOwnershipConflict and never changed or deleted.
Tenant model serving
spec.inference.enabled lets a tenant serve models from its own namespace, reachable only through the internal model gateway. While it is enabled, the operator:
- labels the namespace
bsqai.io/model-serving: enabledand records the gateway in thebsqai.io/model-gatewayannotation, which admits the namespace to the internal model gateway; - applies ingress NetworkPolicies on model pods that admit only the model gateway, Prometheus, the Alloy gateway, kubelet probes, and the model's own pods;
- runs model pods as the
kserve-storageServiceAccount, which has no API token; - when the tenant has private MLflow, writes a read credential for the tenant's model artifacts to the
model-storageSecret, which onlykserve-storagecarries; - keeps the GPU queues, the deployment allowance, and the BSQAI API binding described below.
ModelStorageScoped reports whether that credential reaches only this tenant's artifacts. On Rook-Ceph the operator provisions a scoped read identity. On external S3 it uses an approved scoped <tenantId>-model-s3 Secret in the operator namespace when one exists. Otherwise it falls back to the private MLflow artifact credential and reports False with reason ProviderCannotScope.
When inference is disabled, the operator removes the label, the policies, the credential, and the binding, and withdraws the namespace from the model gateway. A failure in any serving resource also withdraws the namespace from the gateway until the operator reconciles it again.
Deployment allowance
spec.inference.allow lists what the cluster admits in the tenant's serving namespace: allRuntimes, or up to 64 runtimes, and up to 64 resourceProfiles. The operator writes it on the namespace as the annotations bsqai.io/allowed-runtimes, with * for every runtime, and bsqai.io/allowed-resource-profiles. Without allow, the tenant deploys no model.
The tenant-model-deployment ValidatingAdmissionPolicy reads those annotations for every InferenceService and LLMInferenceService created or changed in a tenant namespace. It also holds tenant models to a platform serving runtime: no custom or init containers, volumes, host settings, or explicit resources, and environment variables only as plain values, not references to Secrets or ConfigMaps. Resources come from the resource profile.
Platform administrators set the allowance in the Portal on the tenant's Deployment policy tab. See Set what a tenant may deploy.
Tenant GPU scheduling
When a tenant has inference enabled and the operator's inference.gpuParentQueue value names the platform's parent queue, the operator creates three cluster-scoped KAI Scheduler queues for the tenant. The tenant's GPU guarantee and cap come from spec.inference.gpu.quota and spec.inference.gpu.limit. Both are GPU units and accept fractions: 0.5 is half of one GPU. The limit must be at least the quota. Without a quota nothing is guaranteed, and without a limit there is no cap.
Platform administrators set both in the Portal under Administration > Tenants > the tenant > Quotas, in the GPU allocation card, as Guaranteed GPUs and GPU limit. The same tab's Compute quota card sets the namespace ResourceQuota: CPU requests, Memory requests, CPU limits, Memory limits, and Volume storage (spec.resources). Volume storage caps persistent volume claims only, not buckets or databases.
| Queue | Parent | GPU quota | GPU limit | Priority | Used by |
|---|---|---|---|---|---|
tenant-<tenantId> | inference.gpuParentQueue | the guarantee, or 0 | the cap, or no limit | KAI default (100) | no pods |
tenant-<tenantId>-inference | tenant-<tenantId> | the guarantee, or 0 | no limit | 100 | model serving |
tenant-<tenantId>-training | tenant-<tenantId> | 0 | no limit | 50 | pods with priorityClassName: train |
KAI Scheduler places pods only in leaf queues, so the parent holds the guarantee and cap and no pod uses it directly. The leaves have no limit of their own, so the tenant cap bounds all of the tenant's GPU work together. The inference leaf holds the whole guarantee and has the higher priority. Size a tenant's guarantee for its serving needs. The queues set no CPU or memory bound; the namespace ResourceQuota does. When that quota caps CPU or memory limits, each LLM model counts its pod and its router-scheduler at the limits of its runtime profile: with the default values, a one-GPU model counts 3 CPU and 18 GiB.
The tenant-gpu-queue ValidatingAdmissionPolicy checks every GPU pod in a tenant namespace. A pod counts as a GPU pod when it requests nvidia.com/gpu or carries the gpu-fraction or gpu-memory annotation for a shared GPU. The policy admits it only when:
- the namespace carries
bsqai.io/model-serving=enabled; - the pod does not set
spec.nodeNameand usesschedulerName: kai-scheduler; - its
kai.scheduler/queuelabel names one of that tenant's two leaves, never the bare parent or another tenant's queue; - it sets
priorityClassName: trainwhen it uses the training leaf.
The BSQAI API labels tenant model deployments with tenant-<tenantId>-inference. Global model deployments in the kserve namespace keep default-queue. The queues exist only while inference is enabled.
The operator creates the parent before the leaves. It deletes the leaves before the parent when inference is disabled, when inference.gpuParentQueue is cleared, and when the tenant is deleted. It changes or deletes only queues that carry the tenant's labels and lifecycle annotation. A same-named queue without them is reported as QueueConflict and keeps the namespace out of the model gateway. A same-named leaf without them is never changed or deleted. A tenantId ending in -inference or -training is rejected, because those suffixes name the leaves. With inference enabled, tenantId has at most 46 characters, so tenant-<tenantId>-inference fits a 63-character label value.
Backend access to serving namespaces
The BSQAI API deploys, changes, and removes a tenant's models, and reads their pod logs, only in namespaces where a binding lets it. The platform defines one ClusterRole for that, model-serving-backend, and never binds it cluster-wide. The platform binds it in the Global organization's kserve namespace. The operator binds it in each tenant serving namespace through a RoleBinding named model-serving-backend, whose subject is the ServiceAccount the BSQAI API workload runs as. BSQAI API workers that run under the shared backend account can read KServe resources but cannot change them or read pod logs.
The operator keeps the binding while the tenant has inference enabled. It removes the binding before reconciling anything else when inference is disabled, when the tenant is being deleted, when the operator value inference.backendClusterRoleName is unset, or when inference.backendServiceAccounts is unset or invalid. The operator does not check the role name itself: a set but wrong name keeps a binding that grants nothing. An earlier capability that fails or blocks deletion therefore cannot leave the backend in a namespace it should have left. The operator holds bind on the configured role only.
A RoleBinding named model-serving-backend that the operator did not create is never changed or deleted. The operator reports BackendAccessConflict and keeps the namespace out of the model gateway. It does not retry on its own, because it does not watch RoleBindings and this error waits for a change to the PlatformTenant:
-
Inspect the binding and confirm that nothing else depends on it.
export TENANT_ID=<tenantId> kubectl get rolebinding model-serving-backend -n tenant-$TENANT_ID -o yaml -
Delete it, after saving a copy if something else needs its grants.
kubectl delete rolebinding model-serving-backend -n tenant-$TENANT_ID -
Trigger a reconcile by changing the tenant's
PlatformTenant, for example with an annotation. The operator then creates its own binding and admits the namespace to the model gateway again.kubectl annotate platformtenant $TENANT_ID reconcile-requested-at="$(date +%s)" --overwrite
Tenant Grafana Organizations
When the telemetry query gateway serves tenant routes ([authorizer] enabled in the telemetry-query-envoy component, the default), the operator creates a Grafana Organization named <tenantId> for each tenant. In it, the operator maintains the datasources Loki and Tempo and, while Mimir is enabled, Metrics (UIDs loki, tempo and metrics). Each datasource forwards the user's token to the gateway's tenant routes with the fixed header X-BSQAI-Tenant-ID, set to the tenant's Keycloak Organization UUID. The operator rewrites the datasources on every reconcile, and deletes the Organization, with its datasources, dashboards and memberships, when the tenant is deleted. The global tenant keeps the Platform Organization. The operator does not manage Organization membership.
The operator calls Grafana as a server administrator. The platform renders that user's credentials from the grafana component's [admin] settings into the sealed grafana-admin-credentials Secret in bsqai-system. Grafana makes this user an Admin member of every Organization the operator creates. Set [admin] password (GRAFANA_ADMIN_PASSWORD) to a unique value for each site: the platform does not generate it once, so unless RANDOMIZE_SECRETS is enabled it falls back to the platform-wide default secret. See Secret values. If [admin] user changes, add the new user to the existing tenant Organizations before the operator reconciles them. A password change needs no action.
status.observability.grafanaOrganizationId records the Organization's ID and is the operator's proof of ownership. An existing Organization named <tenantId> without a recorded ID is adopted only when its Loki datasource is already pinned to the tenant's Keycloak Organization UUID, as after a lost status update or an operator rollback. The ObservabilityReady condition reports the result for the current generation. A failure moves the tenant to Degraded:
| Reason | Meaning and action |
|---|---|
GrafanaAPIError | Grafana is unreachable or rejected a call; the message carries the HTTP status. The operator retries with exponential backoff, and the tenant recovers on its own. A persistent 403 means the configured administrator is not a member of the tenant Organization, for example after an administrator login change. |
GrafanaCredentialsError | The grafana-admin-credentials Secret is missing or lacks a valid url, username or password. |
GrafanaOrganizationConflict | An Organization named <tenantId> exists that the operator cannot prove it owns. The operator never changes or deletes it. An operator stop or a timed-out request between creating the Organization and its Loki datasource leaves such an empty Organization behind. Confirm that it holds nothing of value, then delete or rename it in Grafana. |
GrafanaDataSourceConflict | Another datasource in the tenant Organization uses a reserved name (Loki, Tempo or Metrics). Rename or remove it. |
IdentityNotReady | The tenant's Keycloak Organization UUID is not published yet. Resolves when IdentityReady is true. |
On a fresh cluster the operator starts before Grafana, so tenants reconciled in between report GrafanaAPIError until a retry succeeds. Tenant deletion waits while Grafana is unreachable, because the operator deletes the Organization before it releases the finalizer. When the gateway's tenant routes are turned off, the operator stops managing Grafana: existing Organizations stay, and tenants deleted afterwards leave theirs behind.
Independent application resources
Platform operators can provision one application bucket or PostgreSQL database per cluster-scoped PlatformTenantResource. This is an operator resource contract, not a tenant-creation wizard option. Submit it through the installation's authorized infrastructure process.
Use a stable object name for retries and bind it to the existing tenant's exact name, Kubernetes UID, and lifecycle UUID. Resource kind, tenant reference, and requested provider name are immutable. A new application resource cannot adopt a legacy tenant or MLflow bucket/database.
apiVersion: bsqai.io/v1alpha1
kind: PlatformTenantResource
metadata:
name: acme-orders-db
spec:
tenantRef:
name: acme
uid: "<actual-PlatformTenant-UID>"
lifecycleId: "<actual-tenant-lifecycle-UUID>"
kind: PostgreSQL
name: orders
reclaimPolicy: RetainReplace both UUID placeholders before submission. The PostgreSQL provider needs CloudNativePG Database and DatabaseRole reconciliation, verified TLS, and the configured maintenance-database access. The generated owner is not a provider superuser. Application databases are independent of the legacy tenant cnpg.enabled switch.
For kind: Bucket, supply credentialsSecretName for an approved S3 credential in the operator namespace. Its ownership metadata binds it to the tenant lifecycle, tenant UID, resource UID, exact endpoint, and resolved bucket name. Create provider-scoped credentials first; metadata does not narrow an overprivileged key. The provisioner's administrator credentials are never distributed as application credentials.
The initial binding fixes the provider and resource ownership. Changing the installation's provider settings later does not move existing data. Read status.resourceName and the ready connection binding instead of constructing names or falling back to shared credentials.
Retention, rotation, and recovery
| Operation | Behavior |
|---|---|
Retain cleanup | Withdraws workload bindings and new database-login access while keeping provider data, ownership records, and canonical credentials for controlled recovery |
Delete cleanup | Also removes the owned database or bucket contents, including object versions and delete markers; cleanup remains pending if the provider cannot complete it |
| PostgreSQL password rotation | Change only the owned canonical credential's password; mirrors converge and readiness waits for successful authentication |
| S3 credential rotation | Replace the approved scoped source; workload copies update on reconciliation |
| Provider relocation | Requires an explicit data migration; reconciliation does not reassign retained data to another provider |
Removing a Kubernetes Secret does not revoke an externally issued S3 key or terminate an existing database session. Coordinate provider-side revocation when immediate withdrawal is required. Bucket deletion needs version-listing and version-deletion privileges; retention locks can prevent completion.
Back up protected ownership journals and canonical credentials as well as provider data. Retained records intentionally survive namespace removal. Do not delete them as orphan cleanup.
OwnershipRecoveryRequired, MilvusRecoveryRequired, or StorageOwnershipUnverified requires verified original ownership evidence. Restore the exact journal and credential records from backup, repair the reported binding, and let reconciliation retry. For a Milvus recovery marker, restore the original journal before clearing its recovery-required annotation. Never reconstruct ownership from names, attach old records to a new UID, or force-remove a finalizer.
Install the CRDs, RBAC, and operator together. Keep a supporting controller while resource objects or cleanup finalizers remain. The application-resource CRD's Argo CD prune protection does not make direct CRD deletion, Application deletion, or Helm uninstall safe. Complete the intended Retain/Delete lifecycle before removing that control plane.
Interacts With
BSQAI APIandTemporal, which accept and orchestrate tenant lifecycle requests.Keycloak, which stores the tenant Organization, its compatibility groups, and the optional external identity provider broker.CloudNativePG, which creates optional tenant database resources.Rook-Cephor external S3-compatible storage, which stores optional tenant buckets and, on Rook-Ceph, holds the tenant S3 role.MilvusandMLflow, which provide the optional tenant vector database and private tracking server.KServeand the internal model gateway, which serve tenant models from the admitted serving namespace.KAI Scheduler, which schedules tenant GPU pods into the tenant's inference and training queues.MCP Lifecycle Operator, which runs Managed MCP Servers in the prepared tenant namespace.Grafana, which holds each tenant's Organization and telemetry datasources.Kubernetes, through namespaces,ResourceQuota, andLimitRangeobjects that enforce tenant compute boundaries when resource requests or limits are configured. The operator still reconciles empty quota and limit-range objects when no values are set.