Administration and operations

Manage tenant lifecycle and find the controls used to operate BullSequana AI.

Agentic Friendly

BullSequana AI separates platform-wide tenant lifecycle, tenant-specific administration, infrastructure operation, and resource ownership. The portal exposes tenant lifecycle and configuration through a dedicated Administration workspace.

In this section

Administration workspace

The workspace switcher exposes Administration to platform administrators, administrators of at least one tenant, and members whose custom role grants tenant.usage.read. Tenant pages additionally require administrator authority in the selected tenant. The exceptions are Analytics and Budgets, which holders of tenant.usage.read open without administering the tenant; for them these are the only pages in the workspace. Other baseline reads and delegated custom-role permissions do not grant entry to this workspace; individual pages and actions still enforce their operation permissions.

The sidebar has two groups:

GroupPages
PlatformTenants: the tenant registry, tenant creation, each tenant's record and its deletion. Administrators: the platform administrators. GPU usage: what each GPU in the cluster is used for; see Review GPU usage. Configuration: the settings that apply to every tenant; see Platform configuration.
TenantGeneral, Analytics, Budgets, Access Control (Members, Groups, Roles, Service accounts), Collections, Theme, Service Desk, Integrations, Allowed Resource Profiles, Deep Research, and Code sandbox where the agent sandbox is enabled

Each entry appears only to those who hold its permission. Tenant administrators see Tenants too, with only the tenants they administer. A tenant's enterprise sign-in and its administrators are tabs of its record under Tenants, not entries of the Tenant group.

The tenant selector under Tenant in the sidebar lists the Keycloak Organizations the signed-in user administers, and those where they hold tenant.usage.read. Tenant-scoped navigation remains disabled until an organization is selected. Changing the tenant reloads the selected administration page with the new tenant context.

Infrastructure health and GitOps operations remain available through the platform's operational services and the corresponding deployment and Foundation documentation. They are not a separate section in the Administration sidebar.

For access tasks, use Platform and tenant administrators, Manage members, Manage groups, Manage roles and permissions, and Manage service accounts. For exact permission definitions, use the reference catalogue.

Tenant registry

Open Administration → Tenants to inspect tenant lifecycle records. The registry supports:

Administration tenant registry with navigation, tenant filters, status and services columns, and Create tenant action

  • search by display name, slug, or organization identifier
  • filtering by lifecycle status
  • service indicators for Keycloak and CloudNativePG
  • the current lifecycle phase and last update time
  • actions to open or delete a tenant

The registry refreshes while lifecycle operations are running. Above it, platform administrators see the tenants that are over their monthly BCU allowance this month.

Platform administrators see every tenant. A tenant administrator sees only the tenants they administer, without the operator's diagnostic messages and conditions.

Provision a tenant

Tenant creation is asynchronous. The portal submits a lifecycle request and continues to display progress while the platform reconciles the tenant. Creating a tenant requires platform.tenant.create.

  1. Open Administration → Tenants.
  2. Select Create tenant. The wizard has four steps: Basics, Services, Authentication, and Review.
  3. In Basics, enter the immutable lowercase tenant slug and an optional display name. The slug must not belong to an existing or deleted tenant.
  4. In Services, keep Identity and access (Keycloak) enabled. It is required.
  5. Keep Object storage bucket selected unless the tenant needs no file storage. It creates the tenant's own bucket for library files, their extracted content, and sandbox workspaces. Without it, file uploads are unavailable in the tenant.
  6. PostgreSQL database (CNPG) is selected by default. It creates a dedicated PostgreSQL database and credentials for the tenant. Clear it when the tenant needs no application database.
  7. Select MLflow if the tenant's private MLflow should start during provisioning. MLflow stores its artifacts in the object storage bucket, so it requires the bucket: turning the bucket off clears and disables the MLflow choice. The tenant slug must have at most 50 characters. Leave MLflow off to enable it later.
  8. In Authentication, choose local Keycloak accounts or an external OIDC/SAML provider. Enter the provider details when using enterprise sign-in.
  9. In Review, check the configuration and select Start provisioning.
  10. For an external provider, complete the generated callback and credential setup, then test sign-in as described in Enterprise sign-in.
  11. Assign the tenant's first administrators on the tenant record's Administrators tab; see Assign tenant administrators.

The Services step of Create tenant with Identity and access (Keycloak), Object storage bucket, PostgreSQL database (CNPG), and MLflow

The wizard does not enable inference. To let the tenant deploy its own models, enable inference once it is provisioned.

The wizard does not set up the tenant's Data services either: its metastore project, Data Catalog, and Data authorization. A platform engineer adds them in the platform configuration once the tenant is Ready, as Tenant Data services describes.

After the tenant Organization is provisioned, a platform administrator can grant its tenant administrators permission to manage services. This permission is off by default and is separate from the creation request.

Private MLflow requires the installation prerequisites in Configure private tenant MLflow, including artifact storage. Its database is independent of the optional tenant application database.

The object storage bucket is the tenant's <tenantId>-data bucket on the installation's object storage provider, Rook-Ceph or external S3. BullSequana AI stores the tenant's library files, their extracted content, and its sandbox workspaces there, and private MLflow writes its artifacts to the same bucket. The Global organization keeps the platform's own buckets. While a tenant has no ready bucket, its file uploads and sandbox starts fail with 503 tenant_data_binding_unavailable.

The suffixes -inference and -training name a tenant's GPU queues, so a slug ending in either is refused with 400 tenant_gpu_queue_slug. Enabling inference, at creation or later, also needs a slug of at most 46 characters that contains no tenant- segment and does not end in -kserve. Otherwise the request returns 400 tenant_inference_slug. The tenant's GPU queues exist only while inference is enabled. See Tenant GPU scheduling.

Inference also needs the tenant's private MLflow, because the tenant's model servers read model artifacts with MLflow's storage credential. A tenant cannot have inference without MLflow selected, and its MLflow cannot be turned off while inference is on. The rule follows the selected state, not readiness: until the tenant's MLflow is ready, its model deployments fail with 503 mlflow_not_enabled. A tenant that only uses models of the Global organization or models other organizations publish to it needs neither inference nor MLflow.

The provisioning page reports these stages:

  • Provisioning request accepted
  • Tenant namespace
  • Keycloak organization
  • Object storage bucket, when selected
  • PostgreSQL database, when selected
  • MLflow, when selected
  • Tenant ready

Tenant provisioning progress with completed and reconciling lifecycle stages

Provisioning continues after the administrator leaves the page.

Enable inference for a tenant

Inference lets a tenant deploy and serve its own models in its namespace. The Portal offers no switch for it. A platform administrator enables it through the BSQAI API, at creation with services.inference set to true in POST /v1/admin/tenants, or later:

export BSQAI_API_URL=https://<api-host>
export ACCESS_TOKEN=<platform-administrator-access-token>
export LIFECYCLE_ID=<tenant-lifecycle-id>

curl -s -X PUT "$BSQAI_API_URL/v1/admin/tenants/$LIFECYCLE_ID/inference" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled":true}'

The request returns 202 Accepted. Poll the lifecycle record until the tenant is Ready again. The tenant needs MLflow selected and a slug that meets the inference rules in Provision a tenant. Until inference is enabled, a deployment or import in the tenant is refused with 409 model_installer_tenant_inference_disabled, which the Portal shows as "Model serving is not enabled for this organization." {"enabled":false} disables it again.

Members still need the Inference Deployer or Inference Operator role to deploy models; see Inference permissions.

Inspect tenant status

Select a tenant from the registry to open its record. The record has these tabs:

TabInformation
Overviewlifecycle and organization identifiers, desired state, timestamps, and observed and desired generations
Administratorsthe tenant's administrators, direct or inherited from platform administration; see Assign tenant administrators
Authenticationlocal accounts or the tenant's external identity provider; see Enterprise sign-in
ServicesKeycloak and CloudNativePG readiness, the object storage bucket state, plus private MLflow state, permitted actions, and tenant service-management permission
Deployment policythe models, serving runtimes, and resource profiles the tenant may deploy, and whether it may publish its models to all tenants; see Set what a tenant may deploy
Quotasthe tenant's GPU allocation, compute quota, and monthly BCU allowance; see Set a tenant's quotas
Lifecycle healthreported conditions, status, reason, and message; shown only to holders of platform.tenant.read

Tenant lifecycle health with reconciled namespace, identity, storage, and database conditions

Use the lifecycle view when a tenant remains in Pending, Provisioning, Degraded, or Deleting.

A tenant stays Provisioning while it waits for something. The waiting condition is Unknown, and its reason names what it waits for. Some waits need an administrator, so check the reason when a tenant stays Provisioning. A tenant is Degraded when a condition is False. This happens on a failure, such as a permanent error or an unexpected error, or when a capability that was ready breaks.

If NamespaceReady is not True and its reason is NamespaceOwnershipConflict, a namespace named tenant-<slug> already exists but was not created for this tenant. The tenant operator does not change or delete it. Kubernetes cannot rename namespaces, so inspect what the namespace holds, move anything you need elsewhere, and delete it. The operator retries every 30 seconds and provisions the tenant without any change to it.

DatabaseReady is not True until the tenant's application database is closed to other roles. When that isolation check fails, the operator also removes the tenant pods' network path to the database, and restores it once the check passes.

The operator checks again every 30 seconds after you fix OtherRolesConnected or DatabaseIsolationInvalid. It does not retry OwnershipConflict, OwnershipRecoveryRequired or InvalidCA on its own. A missing CA Secret is retried automatically, as a transient error. After you fix one of those, restart the tenant operator with kubectl -n bsqai-system rollout restart deployment/bsqai-tenant-operator, which checks every tenant again.

ReasonCause and fix
OtherRolesConnectedA role that may no longer connect to the tenant database still holds a session there, opened before access was closed. Revoking access does not end it. As a PostgreSQL superuser, list exactly those sessions with SELECT a.pid, a.usename FROM pg_stat_activity a JOIN pg_database d ON d.datname = a.datname WHERE a.datname = '<slug>' AND NOT has_database_privilege(a.usesysid, d.oid, 'CONNECT');. Confirm each is foreign, then end it with SELECT pg_terminate_backend(<pid>);. The owner's and superusers' sessions are not listed, so do not end them. The reason also shows briefly while other sessions of the owner role exit, which the operator ends itself.
DatabaseIsolationInvalidOne of these holds:
• <slug>_owner does not own the database.
• The owner or <slug>_app has superuser, create-database, create-role, replication or bypass-RLS attributes.
• Either role holds or grants a role membership.
• <slug>_app owns the database.
• A grant the operator could not revoke remains.
Restore both roles to plain login roles in PostgreSQL.
OwnershipConflictThe database exists but is not owned by the tenant role. Restore the verified owner before retrying.
OwnershipRecoveryRequiredThe database's resource is recorded but the database itself is missing in PostgreSQL. Restore it from backup, or remove the database capability deliberately.
InvalidCAThe CNPG cluster's <cluster>-ca Secret in the CNPG namespace has no ca.crt, or it is empty or not valid PEM certificate data. Correct the Secret, then restart the tenant operator. A missing Secret is not InvalidCA: the operator reports a transient error and retries every 30 seconds until the Secret exists again.

MLflowReady on the PlatformTenant reports the same reasons for the tenant MLflow database (<slug>_mlflow). The portal shows MLflow condition reasons only as operator_status, so read them with kubectl get platformtenant <slug> -o yaml.

Set what a tenant may deploy

Open Administration → Tenants, select the tenant, then open Deployment policy. Platform administrators set there which models the tenant may deploy, on which serving runtimes and resource profiles, and whether the tenant may publish its models to all tenants. Tenant administrators and members cannot widen the policy.

PartWhat the tenant may use
Modelsevery model, or only the listed Hugging Face repository IDs and registered model names; an empty list allows none
Serving runtimesevery serving runtime, or only the selected ones
Resource profilesonly the selected profiles; with none, the tenant cannot deploy any model

Allow every model and Allow every serving runtime are on by default; turn one off to list the models or select the runtimes. Select Save policy to apply the changes, or Discard changes to drop them.

A tenant's Deployment policy tab with the Models and Serving runtimes switches, the allowed resource profiles, and the Publishing to all tenants card

The policy applies to every model deployment and import in the tenant, from Studio and from the BSQAI API. A request outside it fails, and the error names the allowed values. A model is listed by its Hugging Face repository ID or registered model name; a model deployed from any other location, such as s3://, is matched by its full URL. The policy covers only the tenant's own deployments and imports. It does not limit which models other organizations offer the tenant. Saving a policy stops nothing: existing deployments keep running, and Studio marks the ones the policy no longer allows Outside policy. New deployments follow the saved policy.

A tenant whose policy has never been set may deploy every model on every serving runtime, with the resource profiles that were enabled for it before. Its tenant administrators see the resource profiles in Tenant settings but cannot change them. Platform administrators define GPU-share and multi-GPU profiles there too; see Define GPU profiles.

Let a tenant publish its models

Below the policy, the Publishing to all tenants card has one switch. It saves as soon as you change it, separately from Save policy. Switched on, the tenant's administrators may publish the language models the tenant deploys, so every tenant can choose to use them. The tenant keeps running and paying for its models. Each other tenant decides through its own allow rules which of its members may use them, and sees only the model, never its deployment.

  • The switch applies once inference is enabled for the tenant.
  • Predictive models cannot be published.
  • Turning the switch off unpublishes every model the tenant published. Other tenants' allow rules on those models stay, so they apply again if the model is published later.

Tenant administrators publish a model from its Access sheet in Studio; see Publish a model to all tenants. A platform administrator can then narrow a published model to chosen tenants, or unpublish it, in Model routing. The same operations are in the Model access and publishing API.

Set a tenant's quotas

Open Administration → Tenants, select the tenant, then open Quotas. The tab has three cards. Only platform administrators change them. The site-wide defaults that new tenants start with are in Tenant defaults and usage metering.

A tenant's Quotas tab with the GPU allocation, Compute quota, and Monthly BCU allowance cards

GPU allocation

The GPU allocation card sets Guaranteed GPUs and GPU limit for all of the tenant's model deployments together. Both accept fractions of a GPU, such as 0.25, with at most three decimals. An empty guarantee, or 0, guarantees nothing. An empty limit sets no cap. The limit must be at least the guarantee. The allocation takes effect once inference is enabled for the tenant; see Tenant GPU scheduling.

The platform can guarantee tenants only the GPUs that the KAI Scheduler parent queue guarantees, less what the platform's own queues and the other tenants already hold. Raising a guarantee beyond that is refused, and the error states how many GPUs remain. Lowering a guarantee is always accepted. When the parent queue's GPU quota is unlimited, no guarantee is refused. Without KAI Scheduler or its parent queue, the platform guarantees no GPUs. With the installation defaults, the parent queue reserves no GPUs for tenants, so any Guaranteed GPUs above 0 is refused with tenant_gpu_capacity_exceeded until the platform team reserves some; see Tenant GPU guarantees.

Compute quota

The Compute quota card caps the CPU, memory, and volume storage that the tenant's namespace may use, as Kubernetes quantities such as 8, 500m, or 32Gi:

FieldWhat it caps
CPU requestsTotal CPU that the tenant's containers may request
Memory requestsTotal memory that the tenant's containers may request
CPU limitsTotal CPU limit of the tenant's containers
Memory limitsTotal memory limit of the tenant's containers
Volume storageTotal storage that the tenant's persistent volumes may request. Buckets and databases are not counted.

An empty field sets no cap. Lowering a cap keeps running workloads; new ones beyond it are refused. The card shows Applying until the cluster enforces the saved values, then Applied.

The tenant operator enforces the quota as the namespace ResourceQuota. A container in the tenant namespace that sets no request or limit for a capped resource receives a default from the namespace LimitRange, so it still counts against the cap. By default that is a request of 100m CPU and 128Mi memory and a limit of 1 CPU and 1Gi memory; see Tenant container defaults. New tenants start with the installation's compute quota defaults, which set no cap unless the platform team configures one.

Monthly BCU allowance

The Monthly BCU allowance card sets the tenant's soft monthly limit in BCU. Reaching it refuses nothing. See Set a tenant allowance.

Manage private MLflow

Open Administration → Tenants, select the tenant, then open Services. The MLflow card shows the requested state, observed state, and any conditions preventing readiness. Platform administrators can enable or disable supported services for any tenant. Tenant administrators can do so only when a platform administrator explicitly grants tenant.service.manage for that tenant.

The Tenant service management control grants or revokes this permission for the tenant's current administrators across all supported tenant services. It does not grant MLflow content access. Revocation blocks new requests; previously accepted changes continue. Enabled services remain running. To stop MLflow, submit an explicit disable request.

  1. Select Enable MLflow or Disable MLflow when that action is available. Enable MLflow is not offered for a tenant without an object storage bucket, and Disable MLflow is not offered while the tenant's inference is on.
  2. Wait for the requested change and observed state to converge. Acceptance of a request does not mean the service has finished changing.
  3. If the card shows Needs attention, review the reason and blockers. Refresh the status and use the available retry action. Contact a platform administrator when recovery is required.

Disabling MLflow stops access while retaining its database and artifacts for re-enablement. Re-enabling the same tenant restores its recorded storage location.

When the card shows MLflow ready, the tenant's members use the private MLflow in Studio's Model repository and Experiments and through the BSQAI API. Their AI Viewer or AI Builder role still decides what they can read or change. While MLflow is provisioning, Studio reports that the service is being set up and opens the pages when it becomes ready. In every other state, Studio and the API report that the organization does not have the service enabled. They never show the Global MLflow in its place. Experiments and models in the Global MLflow are not moved into the tenant. The card does not provide a direct MLflow browser link.

Platform configuration

Administration → Configuration holds the settings that apply to every tenant on the platform. Reading it requires platform.config.read; changing it requires platform.config.update.

Its Code sandbox card has one switch, which lets every tenant's members run code and preview apps in the sandbox. It saves as soon as you change it. The card also states that sandboxes can reach the public internet over HTTPS. Turned off, no tenant can use the sandbox, and no tenant administrator can turn it back on. While it is on, each tenant's administrator can still turn the sandbox off for their organization in Tenant settings. Through the API, read and change the setting with GET and PATCH /v1/admin/platform/configuration. Turn the sandbox off without a render describes the effect on running sandboxes.

Administration Configuration page with the Code sandbox card and its switch

Tenant lifecycle control plane

The BSQAI API stores the requested lifecycle state, and a Temporal workflow applies it to the cluster as a PlatformTenant resource. The BSQAI Tenant Operator reconciles that resource. It creates the tenant namespace with its ResourceQuota and LimitRange, the Keycloak Organization with its tenant-administrators group, the tenant database with its roles and tenant-db-credentials Secret on the shared CloudNativePG cluster, the object storage bucket, the private MLflow, the KAI Scheduler GPU queues while inference is enabled, and the tenant's Grafana organization where the operator manages Grafana. The operator reports conditions on the resource. The workflow reads them back, and the BSQAI API returns them as the lifecycle record's conditions and per-service status.

The tenant administration API is available under /v1/admin/tenants:

RoutePurpose
POST /v1/admin/tenantsAccept a tenant provisioning request
GET /v1/admin/tenantsList lifecycle records. Tenant administrators receive only the tenants they administer, without diagnostics.
GET /v1/admin/tenants/{lifecycle_id}Read conditions and service status
DELETE /v1/admin/tenants/{lifecycle_id}Accept a tenant deletion request
PUT /v1/admin/tenants/{lifecycle_id}/inferenceRequest inference enablement or disablement with {"enabled":true} or {"enabled":false}
PUT /v1/admin/tenants/{lifecycle_id}/inference/gpuSet the tenant's GPU guarantee (quota) and cap (limit), as a platform administrator
PUT /v1/admin/tenants/{lifecycle_id}/inference/publishingLet the tenant publish its models to all tenants, with {"enabled":true}, or stop it with {"enabled":false}, as a platform administrator
PUT /v1/admin/tenants/{lifecycle_id}/compute-quotaSet the tenant namespace's CPU, memory and volume storage caps, as a platform administrator
GET, PUT /v1/admin/tenants/{lifecycle_id}/deployment-policyRead or replace the tenant's deployment policy, as a platform administrator
GET, PUT /v1/admin/tenants/{lifecycle_id}/authenticationRead or change the tenant's sign-in configuration, as its tenant administrator or a platform administrator
GET /v1/admin/tenants/{lifecycle_id}/authentication/groupsList the tenant groups that OIDC group synchronization may target
GET /v1/admin/tenants/{lifecycle_id}/servicesRead managed service state and permitted actions
GET /v1/admin/tenants/{lifecycle_id}/services/{service}Read one managed service
PUT /v1/admin/tenants/{lifecycle_id}/services/{service}Request service activation or disablement
GET, PUT /v1/admin/tenants/{lifecycle_id}/services/managementRead or change the tenant-wide service-management permission, as a platform administrator

Administrator assignments have their own routes; see Administrator API.

Create and delete return 202 Accepted. Poll the lifecycle record instead of treating the initial response as completion. A create request returns 409 Conflict when the slug belongs to an existing tenant or to a tenant that is deleted or being deleted. A create request returns 400 tenant_gpu_queue_slug for a slug ending in -inference or -training. A create request with services.inference set to true, or an inference enablement request, returns 400 tenant_inference_slug when the slug does not meet the inference rules in Provision a tenant.

The GPU allocation body is {"quota":"0.5","limit":"2"}, with null for no guarantee or no cap. A limit below the quota returns 422. A quota raised beyond what the parent queue can still guarantee returns 409 tenant_gpu_capacity_exceeded.

The compute quota body is {"requests":{"cpu":"8","memory":"32Gi"},"limits":{"cpu":null,"memory":null},"storage":"500Gi"}, with null for no cap. Each value is a Kubernetes quantity. The request returns 202 Accepted. The tenant record then holds compute_quota and compute_quota_applied, which is false until the namespace enforces the saved values.

The deployment policy body is {"models":[...],"runtimes":[...],"resource_profiles":[...]}. null for models or runtimes allows every one, and an empty list allows none. Every listed serving runtime and resource profile must exist on the cluster. GET /v1/resource-profiles/selectable, which requires platform.global.read, lists the profiles the cluster offers. The policy cannot be set before the tenant Organization is provisioned. Members read their active tenant's policy with GET /v1/deployment-policy, which requires tenant.read. A platform administrator acting in a tenant, the Global organization included, replaces its policy with PUT /v1/deployment-policy.

A deployment or import outside the policy returns 400 deployment_not_allowed, whose param is model, runtime, or resource_profile, and 409 deployment_no_resource_profile when the tenant has no resource profile. Model presets refuse a resource profile outside the policy in the same way. GET /v1/model-installer/deployments reports outside_policy: true for a deployment the current policy no longer allows.

The managed service identifier is mlflow. The creation request uses services.mlflow as a boolean, and services.storage, which defaults to true, selects the object storage bucket. Activation requests return 202 Accepted; poll the service state for completion.

The API enforces the dependencies between these services:

RequestRefused with
Create a tenant with services.mlflow set to true and services.storage set to false400 tenant_service_invalid
Enable MLflow with PUT /v1/admin/tenants/{lifecycle_id}/services/mlflow for a tenant without storage400 tenant_service_invalid
Create a tenant with services.inference set to true and without services.mlflow400 tenant_service_invalid
Enable inference while the tenant's MLflow is not selected409 tenant_service_conflict
Disable MLflow while the tenant's inference is on409 tenant_service_conflict

The management endpoint returns {"permission":"tenant.service.manage","granted":false} when permission has not been granted. Its PUT body is {"granted":true} to grant or {"granted":false} to revoke. Only platform administrators may use this endpoint. Changing the grant requires the tenant Organization to exist. A tenant administrator cannot grant this permission through a custom role.

A tenant-bound API key can read service state with an explicit tenant.read scope and change service state with an explicit tenant.service.manage scope. The key owner must still have the required authority in that tenant. Keys cannot create tenants or grant service-management permission. The API checks the key, its tenant binding and scope, and the owner's current authority when accepting the request. Later key revocation, loss of owner authority, or management-permission revocation blocks new requests; previously accepted changes continue.

Delete a tenant

Deleting a tenant removes the tenant and its data, and cannot be undone. Back up or export tenant data that must be kept before requesting deletion.

  1. Open the tenant's action menu in Administration → Tenants.
  2. Select Delete.
  3. Type the immutable tenant slug to confirm.
  4. Monitor the lifecycle record until deletion completes.

Deletion runs in this order:

  1. BullSequana AI stops the tenant's running work: the schedules of its data pipelines, and the workflows started for it, such as file ingestion, reindexing, deep research and Skill publication. A removal of library files that the tenant stored before it had its own bucket is not stopped, but left to finish.
  2. It removes its organization's Connectors: Managed Server workloads and their environment, Remote Server registrations, and every member's linked credential for them.
  3. The tenant operator removes the tenant's platform resources: its namespace and the workloads in it, its application database, its object storage bucket with the library files, sandbox workspaces and MLflow artifacts in it, its private MLflow with its metadata database, its Keycloak organization, its Grafana organization and its GPU queues.
  4. Once they are gone, BullSequana AI removes the tenant's models from model routing: their routes, the connections that no other route uses, and the allow rules that any organization set on them. It also removes the tenant's own allow rules on models of other organizations and of the Global organization, and takes the tenant off every model's list of organizations it is offered to. Other organizations lose access to the models the tenant offered them.
  5. Once the steps the stopped workflows were running have ended or reached their time limit, and any removal of library files has finished, it removes what the tenant stored on the platform's shared services: chats with their messages and stored responses, library records, agents, Skills and their packages, model presets, provider credentials such as web-search API keys, service accounts and their Keycloak clients, data connectors, branding, translations and settings; its knowledge-base collections; the files it stored in the platform's buckets before it had its own bucket, and the copies of them that moving them to its bucket left behind; and the authorization records of the organization, its groups and its custom roles.
  6. The deletion is recorded.

If a step fails, the tenant stays Deleting and the step is retried until it succeeds. The lifecycle record shows the reason; failure type DataPurge means stopping the tenant's work (step 1) or removing its data from the shared services (step 5) failed.

What deletion keeps

KeptFor how longHow it is removed
The tenant's lifecycle record: slug, display name, lifecycle ID and deletion timePermanently. It keeps the slug reserved: a replacement tenant needs a new slug.It is not removed. The registry no longer lists the tenant.
Usage records of the tenant's model calls and API requests, and its capacity BCU recordsNo automatic expiry, so past usage and BCU statements stay correct after deletionA platform operator deletes them from the BSQAI API database by the tenant's organization ID, in the usage_invocations, usage_api_requests and usage_capacity tables.

Usage and capacity records hold no prompts, responses or files. Audit and compliance evidence lists what a usage record holds.

Deleting a deleted tenant again with DELETE /v1/admin/tenants/{lifecycle_id} repeats steps 1, 2, 4 and 5, and removes what is left of its data. Use it for tenants deleted with an earlier version, which removed only their platform resources. Their private MLflow metadata databases stay; Private tenant MLflow describes how to remove them.

Responsibility boundaries

RolePrimary responsibility
Platform administratortenant lifecycle, deployment policies, quotas and allowances, model routing and Global models, platform administrators, GPU usage, and the platform configuration
Tenant administratormembers, groups, roles, service accounts, tenant settings, integrations, branding, user and group caps, and the tenant's administrators, inside one tenant
Model and resource ownerswho may use a model they deployed, with its Grant Managers, and the sharing of the experiments, models, files, and Agents they own
Inference Deployer or Inference Operatordeploying and operating the tenant's own models
Platform engineerinstallation, reliability, capacity, upgrades, recovery, and lifecycle automation

A platform administrator does not administer every tenant: tenant administration needs its own assignment, except the inherited administration of Global. Use Security and compliance for control boundaries and multi-tenancy and tiered RBAC for Data workspace behavior.

Observe platform behavior

Foundation collects metrics, tenant-routed logs, traces, and operational events. Operators use these signals to inspect:

  • tenant lifecycle and reconciliation failures
  • request volume, latency, saturation, and errors
  • model-serving and accelerator capacity
  • database, object-storage, and block-storage health
  • authentication and authorization failures
  • tenant-scoped activity and durable audit events

Platform administrators also see how GPUs are used in Review GPU usage, and each tenant's consumption in BCU in Review tenant usage. Start with Observability and audit, Grafana, and Troubleshooting.

Upgrade and recover

Use the release notes to select the deployment path before changing a cluster. Modified environments require an explicit compatibility review and a rollout plan for the target upgrade path.

See Upgrade 1.4.0 to 2.0.0 and PostgreSQL backup and disaster recovery.

On this page