Upgrade and release process

This guide is for platform engineers upgrading an existing BullSequana AI 1.3.0 to 1.4.0 installation. It separates the managed upgrade from the manual procedure for modified deployments. Historical upgrade guides are linked at the bottom.

Choose the upgrade path

The managed baseline is deployment-specific. Original approved hostnames, sizing and provider options are part of it; later template edits, chart replacements, topology changes and unclassified live drift are not. A cluster reporting version 1.3.0 or showing green Applications is not sufficient evidence. The CLI's --accept-risk option does not make a modified deployment supported; do not use it for the managed procedure on this page.

Prepare the maintenance window

Complete this shared preparation before starting either upgrade path:

1. Select and record the exact 1.4.0 platform source commit and matching image/chart artifacts. Use a separate target checkout with VERSION equal to 1.4.0. Keep the original 1.3.0 checkout and GitOps commit intact. Confirm the target CLI exposes upgrade record-baseline and the 1.3.0 to 1.4.0 contract; an older installed CLI is not interchangeable.
2. Confirm access to the intended Kubernetes context, Argo CD, manifests repository, registry, PostgreSQL, object storage and Keycloak administrative recovery account. The workstation needs Python 3.12, uv, kubectl, Helm, kubeseal and Git. The manual path also uses the Argo CD CLI authenticated to this installation.
3. Allow a restarted cluster to recover fully. Check nodes, workloads, storage, recent OOMs, failed hooks and application-level access. Resolve baseline health problems before upgrading. Reserve a maintenance window; this procedure does not promise zero downtime.
4. Preserve platform.env, local.env, generated_secrets.env, the active sealing key/certificate, GitOps revisions and Argo CD Application/AppProject policies. Back up databases, object data, volumes and operator ownership journals. Test recovery access. The CLI's logical database and metadata snapshots do not replace full application-consistent backups.
5. Record representative existing data and access: tenant/user IDs, memberships, OpenFGA store IDs and unrelated sharing grants; an existing file and its checksum; an MLflow run/artifact; a Milvus collection/query; and a chat/inference check if a model is available. Include a non-Global tenant where used. Keep private test evidence out of ordinary logs.
6. Complete Airbyte extraction and cutover preparation before permitting retirement. Inventory administrator assignments and appoint the platform administrator who will remap them. The CLI neither transfers Airbyte connections nor chooses replacement permissions.
7. Stop scheduled ingestion and other writers as required for consistent backups and component migrations. Coordinate tenant provisioning/deletion, uploads, training jobs and schema changes during the window. Keep Rook, CNPG, CSI and other lifecycle controllers running when they are needed for migration or deletion.

Keep recovery files in a private directory with owner-only permissions (0700 directories, 0600 files). Follow the installation's encryption and retention policy. Do not put credentials, dumps, kubeconfigs or unseal keys in Git, a PR, chat, or CI logs.

See Backup and disaster recovery for the recovery plan.

Preserve deployment configuration and credentials

Install dependencies in the target checkout and use that checkout's CLI throughout:

uv sync --locked
uv run -m src.cli.main upgrade plan --from 1.3.0 --to 1.4.0 --offline

The offline command only prints the contract. It does not prove live eligibility.

Bring the existing private deployment inputs into the target checkout through your approved secret-handling process. Reuse these exact sealing files:

src/manifests/seal/sealed-secret.crt
src/manifests/seal/sealed-secret.key

Do not run apply-sealing-key, initialize OpenBao again, or install a replacement Argo CD. Do not generate replacement passwords to work around missing original credentials. Recover them from their authoritative source first, including database passwords, OIDC client/cookie secrets, S3 credentials, the LiteLLM master key, MLflow service credentials and encryption keys such as Superset's secret key and Lakekeeper's secret-backend key.

Keep the same storage/secrets providers, bucket and database identities, DNS zone and public hostnames. A provider migration or hostname change is separate work. In particular, workspace.<domain> is the new Kubeflow default, not an instruction to rename an existing deployment's endpoint. If the current host is kubeflow.<domain>, preserve it in the target inputs:

export KUBEFLOW_COMPONENT_URL_PREFIX=kubeflow

Use full TOML-path override names, including COMPONENT where applicable. Verify the rendered routes, redirect URIs, storage bindings and resource sizing against the installation before publishing. If Azure DNS is enabled, the resource group embedded in AZURE_DNS_CLIENT_JSON must match AZURE_DNS_RESOURCE_GROUP_NAME.

Managed upgrade

Complete the shared preparation before starting this path.

1. Verify the original installation baseline

The supported 1.3.0 source is commit 868d8abef8afd427f240e360e6fe20e2dfc1348a. An advancing release/1.3.x branch is not equivalent. Use the independently approved installation receipt if one already exists. Otherwise reproduce it from the original source and deployment manifests, not from an edited copy of the current cluster:

export SOURCE_130=/secure/checkouts/platform-1.3.0
export MANIFESTS_130=/secure/checkouts/original-deployment-manifests
export UPGRADE_RECORDS=/secure/upgrade-1.3.0-to-1.4.0
umask 077
mkdir -p "$UPGRADE_RECORDS"
export BASELINE="$UPGRADE_RECORDS/baseline-1.3.0.json"

uv run -m src.cli.main upgrade record-baseline \
  --source-checkout "$SOURCE_130" \
  --manifest-checkout "$MANIFESTS_130" \
  --output "$BASELINE"

Replace the example paths. The original source checkout must contain its private deployment inputs and sealing material; both source and manifest checkouts must have no tracked edits. Pin the manifests checkout to the original installation commit. Add --gpu only if that installation included the platform GPU Operator.

Recording independently renders the source, compares the deployed content and bootstrap releases, and checks the live Application inventory. It writes a new local receipt; it does not bless arbitrary current state or overwrite an existing receipt. Keep the receipt with the installation record. If reproduction fails, investigate the difference or use the manual path. Do not edit the receipt or change the cluster just to silence the check.

2. Run the live plan and dry run

In the target checkout, GIT_BRANCH must still identify the current 1.3.0 manifests branch. The managed upgrader creates a separate immutable target ref itself.

uv run -m src.cli.main upgrade plan \
  --from 1.3.0 --to 1.4.0 --baseline "$BASELINE"

uv run -m src.cli.main upgrade apply \
  --from 1.3.0 --to 1.4.0 --baseline "$BASELINE" --dry-run

Require every check to pass without --accept-risk. The plan checks original provenance, inventory/content, health, idle operations and existing upgrade state. The live dry run also imports known active credentials, renders the target, compares Secret data and public routes, and validates migration prerequisites. It can write ignored local output and generated values; it does not push Git or mutate Kubernetes.

Inspect the render and credential-continuity result. Investigate any missing active source rather than assuming a generated replacement is safe. Confirm Airbyte is absent from the target and the Milvus legacy-coordinator cleanup hook is present. For an ambiguous tenant/store mapping, provide --tenant-store-map /secure/tenant-store-map.json to both dry run and apply. Its JSON maps each target Organization name to its existing legacy tenant ID, for example {"Global":"dataplatform"}. Verify it against Keycloak Organizations and the existing Tuples configuration. It preserves store identity; it does not migrate users or grant privileges.

3. Run durably and monitor with an agent

Recommend running with a platform operator and an AI agent together. Give the agent the approved plan, selected cluster, pinned checkout and redacted status/log access. Let it monitor Argo CD operations, migration Jobs, controller events and application checks. Do not send it raw recovery files or credentials through chat.

Run from a persistent operator session such as tmux, or an equivalent supervised runner that survives a lost SSH/chat connection. Do not launch a second apply while the first still runs.

tmux new -s bsqai-upgrade-140
# Inside tmux: return to the target checkout and restore the same environment.
# BASELINE is the absolute receipt path established above.
uv run -m src.cli.main upgrade apply \
  --from 1.3.0 --to 1.4.0 --baseline "$BASELINE"

Detach with Ctrl-b d; reconnect with tmux attach -t bsqai-upgrade-140. The default operation timeout is 3,600 seconds per Application, not for the whole upgrade. Add --timeout <seconds> only when the maintenance plan calls for it.

The CLI publishes one create-once manifest ref, preserves the existing Argo CD control-plane credentials, saves reconciliation policies and holds an upgrade guard. It snapshots recovery material, drives ordered GitOps operations and executes scoped migration actions. Checkpoints live in argocd/bsqai-platform-upgrade; sensitive recovery files live under artifacts/upgrades/<cluster-uid>/<run-id>/ in the target checkout. Keep that directory, its checksums, inputs and checkout available for resume.

From a second terminal using the same checkout and Kubernetes context:

uv run -m src.cli.main upgrade status
kubectl -n argocd get applications
kubectl get nodes

Do not start competing manual syncs, change target manifests, or remove the upgrade guard while apply is running. Give normal controller retries, pod starts, volume attachments and metrics warm-up time to converge. telemetry-query-envoy depends on services deployed later; its intermediate health can be deferred, but its final health cannot be waived.

4. Handle an unexpected stop

The agent should distinguish slow progress from a persistent environment-specific lock or failed dependency using events, current logs and the operation's elapsed time. A transient backend rollout is not a reason to interrupt a healthy run.

If an intervention is necessary:

1. Confirm whether the original process is still running. Capture upgrade status and the failing Application/Job without exposing secrets.
2. Stop the active apply gracefully if a repair would otherwise race it. Preserve its checkpoint, recovery artifacts and upgrade guard.
3. Diagnose the exact owner and blocking resource. Use the component-specific recovery guidance below. Never clear all PostgreSQL sessions, restart a Kafka quorum together, strip storage finalizers or delete a checkpoint to make progress.
4. Apply the smallest reviewed repair, verify it, then rerun the same apply command from the same checkout and inputs. Resume uses the existing snapshot and completed checkpoints. If target manifests must change, review the correction and use --supersede-failed-snapshot only on a failed, still-locked run. That publishes a new snapshot; it is not an ordinary retry and cannot be combined with --manifest-ref.
5. Record the intervention and resume in the handoff. A recovered upgrade is not an uninterrupted automatic run.

An environment-specific failure is not permission for the agent to remap permissions, replace credentials or broaden access. The operator remains responsible for consequential recovery decisions.

5. Confirm technical completion

Require upgrade status to report completed, with the expected immutable revision. Verify that saved reconciliation policies were restored and the temporary guard is gone. Then run the validation checklist. Technical completion and administrator authorization follow-ups are separate outcomes; report both.

Manual upgrade

Use this path for non-baseline installations. It is a controlled GitOps migration, not upgrade apply --accept-risk and not a fresh installation. The operator performs and records each phase below, adapting resource names only after checking actual ownership. Do not run the managed CLI concurrently. For takeover of a failed managed run, preserve its evidence and do not fabricate a completed checkpoint.

1. Build a site-specific target and recovery record

Complete the shared preparation before starting this path. Compare the original 1.3.0 deployment with the target 1.4.0 render and list every local difference. Preserve deployment-specific tuning. Select a new unused GitOps ref and render all parent and child sources against that same ref:

# Manual path only: use a new target ref, not the active deployment branch.
export GIT_BRANCH=upgrade/site-1.4.0-reviewed
uv run -m src.cli.main validate
uv run -m src.cli.main apply all -r

Review src/manifests/output/ before publication. apply all -r is a renderer, not the managed upgrade: it does not import every live credential or perform continuity gates for you. Compare existing private inputs and authoritative Secrets with the target inputs before accepting newly generated values. Require unchanged active passwords, keys and storage identities; distinguish retired fields from missing active credentials.

Record the target Application inventory and component sync waves. Inspect the Helm-rendered workload order as well as parent waves. In this transition, RAG workers must be compatible before the BSQAI API switches to the new file contract. The release's Milvus cleanup Job must run before its new coordinators. Do not combine a storage provider, secrets backend, DNS or tenant identity migration with this rollout.

Prepare a private backup record. For the released CNPG topology, this captures a logical database backup without printing it:

export UPGRADE_RECORDS=/secure/upgrade-1.3.0-to-1.4.0
umask 077
mkdir -p "$UPGRADE_RECORDS"
kubectl -n argocd get applications,appprojects -o yaml > "$UPGRADE_RECORDS/argocd-before.yaml"
export PG_PRIMARY=$(kubectl -n cnpg get cluster cnpg-cluster -o jsonpath='{.status.currentPrimary}')
kubectl -n cnpg exec "$PG_PRIMARY" -c postgres -- pg_dumpall -U postgres \
  > "$UPGRADE_RECORDS/postgres.sql"

Check exit codes, backup completeness and restoration procedure. Include mlflow, mlflow_auth, Keycloak, OpenFGA and application metadata. Record Kubernetes resource UIDs, PVC/PV and bucket bindings, OpenFGA store mappings and secret fingerprints. Take consistent object/volume backups as required; the SQL dump alone does not protect them.

2. Hold reconciliation and publish the target

1. Export the existing automated policies for the root, parents and children, plus the AppProject's sync windows. Wait for current operations to finish.
2. Add a temporary deny sync window for the platform project, keeping existing windows. Permit manual sync for the planned operations. Pause automated sync on the root, parents and children. A pause on the root alone does not pause its children.
3. Keep automatic reconciliation paused in the staged root/parent/child Application definitions, so a parent sync cannot turn it back on midway. Keep the project guard effective throughout. Record the intended final policies for restoration.
4. Confirm the new GitOps ref is unused, then publish the reviewed render with uv run -m src.cli.main push-manifests "Upgrade BullSequana AI to 1.4.0". Record the resulting commit and freeze that ref. Do not overwrite the active 1.3.0 ref or advance the frozen target during this run.
5. Check the existing Argo CD chart/application against the target. If a control-plane update is needed, Helm-upgrade the existing release with reviewed values, preserving its admin hash, Redis credential, OIDC client, repository configuration and root's 1.3.0 source/version until control-plane health is restored. Do not blindly run the target's fresh helm-apply argocd command, which can replace root settings prematurely. Do not retarget or sync the installation until the Airbyte retirement preparation in step 3 is complete.

3. Prepare Airbyte deletion before any pruning

Airbyte and oauth2-proxy-airbyte are removed in 1.4.0. Complete extraction first. Before syncing a component that owns their resources, inventory and prove the dedicated ownership of:

• namespaces airbyte and oauth2-proxy-airbyte, with their PVCs and bound PVs;
• CNPG Database resources cnpg-airbyte-database, cnpg-airbyte-temporal-database and cnpg-airbyte-temporal-visibility-database, and DatabaseRole cnpg-airbyte-role;
• cnpg/cnpg-cluster-app-airbyte, the airbyte/airbyte-s3 ObjectBucketClaim, its bound ObjectBucket and dedicated Ceph user when Rook provides storage.

Reject shared namespaces, rebound claims or databases owned by another service. Record UIDs and actual SQL/bucket names, not names alone. Remove only Prune=false / Delete=false guards on those verified resources, preserving other annotations. Set their deletion policies before deletion: CNPG databaseReclaimPolicy: delete and databaseRoleReclaimPolicy: delete, dedicated PV persistentVolumeReclaimPolicy: Delete, and ObjectBucket reclaimPolicy: Delete. Wait for CNPG's cnpg.io/deleteDatabase / cnpg.io/deleteRole finalizers to be installed. Do not remove them.

For Rook, verify the claim's StorageClass already uses Delete. A generated ObjectBucket policy alone can be overwritten by its controller. Do not change a shared StorageClass to delete one bucket; resolve a nonstandard ownership/policy setup before proceeding. External-S3 retirement requires the storage owner to remove only the dedicated Airbyte bucket/credentials under that provider's procedure.

Keep Keycloak and the Airbyte OAuth proxy namespace available for its PostDelete cleanup Job. Ensure that namespace still has the correct ca-bundle and keycloak-enroll-credentials dependencies from bsqai-system before pruning the proxy. Keep them private and available until cleanup finishes.

4. Upgrade foundation and inference prerequisites

Retarget the existing root and parent Git sources to the reviewed target. Use controlled parent/selective syncs to create or update child Application definitions without launching all children at once. Ensure every Git source, including Helm $values sources, resolves to the recorded target commit. Keep the final platform version annotation unchanged until verification.

Use resource selection only to stage the Application hierarchy. Run full syncs for component Applications so their hooks execute. For an individual component, after its prerequisites are ready:

export COMPONENT=cnpg-operator
argocd app sync "$COMPONENT" --prune
argocd app wait "$COMPONENT" --sync --health --operation --timeout 3600

Do not wait for an entire parent's health before syncing the children that make it healthy. Handle the specific telemetry health dependency below separately.

Sync components individually, using the target waves within each dependency group:

1. Sealed Secrets, cert-manager, trust-manager and Gateway API. Verify the gateway data plane and existing routes.
2. CNPG operator, its Barman plugin and the CNPG configuration. Verify the DatabaseRole CRD, plugin registration, existing cluster identity, databases and both transaction/session poolers. Restart the CNPG operator only if needed to load the changed plugin, and wait for readiness. Airbyte database/role pruning must now honor the deletion policies from step 3.
3. Rook operator, CSI drivers and Ceph cluster configuration. Let controlled restarts finish. Require healthy storage before continuing; do not restart all storage pods or delete bound claims. Observe the target Ceph version and HEALTH_OK.
4. Keycloak/bootstrap identity and the tenant operator's CRDs/RBAC/controller before dependent tenant functionality. Preserve original users, memberships, controller journals and credentials. Canonical group provisioning is not automatic migration of existing users' privileges.
5. Other foundation services, including OpenBao replacement when enabled and monitoring. Investigate OOMKilled pods using observed workload size and declared resource overrides; do not treat an OOM as a successful warm-up.
6. Inference CRDs, Envoy AI Gateway controller, KAI Scheduler, KServe gateways/configuration and controllers before dependent serving workloads. Preserve deployed model resources and storage bindings.

telemetry-query-envoy can depend on the later BSQAI API authorizer and Loki/Tempo grants. Stage its target resources, continue the required dependencies, then return to its health check. This is a named dependency deferral, not permission to skip failed hooks or final health checks.

Before the Rook/Kubeflow secret producer changes, inspect kubeflow/kfp-s3-credentials and kubeflow/minio-ca. If these authoritative sources carry Reflector mirror bookkeeping (reflects, reflected-version, auto-reflects under reflector.v1.k8s.emberstack.com/), remove only that self-reflection bookkeeping. Preserve Secret data and intended reflection permissions. Do not delete the source credentials.

5. Make Kafka ready, then migrate Milvus

Sync Strimzi/Kafka before Milvus even if the Applications sit in different parent waves on the source installation. Require the current Kafka and milvus KafkaUser generations to be Ready, then prove an actual SCRAM/TLS topic-list connection with the Milvus identity. Kubernetes pod readiness alone does not prove Kafka controller registration or authentication.

If Kafka 4.1.1 remains stuck, inspect current ControllerRegistrationManager logs. The classified failure has a registration request timing out followed by repeated waiting for the previous RPC, with the controller API still failing. A historical startup message followed by success is not a reason to restart. A single-pod recovery is appropriate only after verifying the same cluster identity, at least three Ready controllers, the complete expected voter set, zero follower lag and that the single affected pod is the current leader. Replace only the affected original pod, then verify registration and quorum again. If these conditions are not met, stop for Kafka-specific recovery; never delete Kafka PVCs or cycle the quorum together.

Before Milvus's cleanup hook runs, quiesce its writers and back up etcd plus the associated object data. For the released topology:

kubectl -n milvus exec milvus-etcd-0 -c etcd -- \
  etcdctl snapshot save /tmp/upgrade-140-etcd.db
kubectl -n milvus exec milvus-etcd-0 -c etcd -- \
  etcdctl snapshot status /tmp/upgrade-140-etcd.db -w json
kubectl -n milvus exec milvus-etcd-0 -c etcd -- \
  cat /tmp/upgrade-140-etcd.db > "$UPGRADE_RECORDS/milvus-etcd.db"

Verify the snapshot contains keys and protect the local copy. Confirm the source is Milvus 2.5.16 and the reviewed target includes chart 5.0.22's release-specific milvus-legacy-coordinator-cleanup Job. Run a full milvus sync. The hook removes obsolete milvus-datacoord and milvus-indexnode Deployments before the 2.6 workloads; this avoids their exclusive etcd session blocking the new coordinator. Do not replace this with volume or etcd deletion. Verify the old Deployments are absent, the target workloads and Kafka connection are ready, and an existing collection can be queried.

6. Upgrade the AI services

1. Sync OpenFGA and Temporal with their existing database/store identities. Check their schema/initialization hooks.
2. Sync MLflow using the MLflow requirements. Preserve mlflow and mlflow_auth, S3 credentials and service-account identity. Confirm the official chart's init migration completes and an existing artifact can be read.
3. Check LiteLLM's old advisory lock, then sync LiteLLM using the new session pooler or direct-primary connection, not the old transaction pooler. Verify migration completion and an inference request.
4. If llm-backend/wait-for-mlflow-sa fails with an immutable Job-template error, verify the existing Job is Complete, save its manifest and remove only that completed Job. Let the owning Application recreate it. Do not delete an active or failed Job without diagnosing it.
5. Sync the BSQAI API with its workers and migration hooks. Check the rendered RAG-worker-before-API order, worker readiness, database migration completion and API readiness. Then sync Portal and remaining AI components. Preserve existing file records, objects, chats and user IDs; verify a file read before restarting writers.

7. Preserve authorization stores and upgrade Data services

Before syncing Tuples, preserve the existing mapping in tuples/tuples-openfga-store-id. The old single-tenant default uses store-id-dataplatform; the target uses store-id-<Global-Organization-UUID>.

1. Read the legacy tenant IDs from tuples/tuples-config privately and resolve each target Organization's UUID in Keycloak. For custom tenants, establish an explicit one-to-one mapping; do not infer ownership from a similar name.
2. Copy each existing store ID value to its new store-id-<Organization-UUID> key, retaining other entries. If the target key already points to a different store, stop. Do not create a new store, merge stores or assign users to repair this mismatch.
3. Save the mapping before and after, then sync Tuples and check model/tuple publication into those same stores. Separately hand off authorization remapping to the administrator. Expected model changes do not permit unrelated grant loss.
4. Sync Lakekeeper before Kubeflow, preserving its encryption key, metadata database, warehouse identity and S3 configuration.
5. Sync Kubeflow and verify the original public route, SSO, Profile ownership and existing notebook/pipeline workspace. Confirm kfp-s3-credentials and minio-ca exist after the producer Job completes. If they disappeared through old self-reflection, correct the source annotations and rerun the owning credential producer with the same identities; do not generate new S3 keys. A dashboard shell with an onboarding overlay is not a working workspace.
6. Drain or finish jobs controlled by the old Spark operator. Inventory old CRDs and adapt workload manifests to the Apache Spark operator's API before submitting them. Do not delete old CRDs while they still contain customer workloads. Sync the new operator, then validate a representative Spark job.
7. Sync the remaining Data components, including managed ingestion, Argo Workflows/Events and Superset. Superset's old local-admin bootstrap fields are deliberately removed; verify OIDC access instead of regenerating that administrator password.

8. Complete retirement and verify the target

Allow GitOps to prune airbyte and oauth2-proxy-airbyte and wait for the OAuth proxy's PostDelete cleanup to succeed. Only then remove orphaned dedicated Airbyte namespaces. Keep provider controllers running until their finalizers complete. Remove any temporary cleanup-only credential copies after their consumer has finished.

Verify absence of the recorded Airbyte Applications, namespaces, PVC/PVs, database/role CRs and Secrets. Also verify the physical SQL databases/role and Rook bucket/user are absent using successful database/RGW inventory queries. An error querying a bucket is not proof of deletion. Verify the shared PostgreSQL/Ceph instances and unrelated customer destination data are unchanged.

Revisit telemetry and every deferred health check. Require the exact target Application inventory, target Git revisions, ready workloads, successful hooks, bound surviving storage, healthy PostgreSQL/Ceph and application continuity from the checklist below.

9. Restore normal reconciliation and hand off

Restore each saved automated policy, including its pruning/self-healing settings; remove temporary health overrides. Reconcile reviewed final Application definitions so those policies will survive later parent syncs. Record that final GitOps revision as well as the staged migration revision. After technical gates pass, set the argocd/platform Application's bsqai.ai/platform-version annotation to 1.4.0 and preserve that value in its reviewed desired definition. Remove only the temporary upgrade sync window last, preserving prior windows.

Resume scheduled work according to the maintenance plan. Record manual actions and any remaining administrator tasks. If a technical gate fails, keep the effective reconciliation guard, preserve recovery evidence and resolve that failure before declaring technical completion. Reverting a Git ref or image does not undo a database migration or restore deleted groups, buckets or volumes.

OpenBao OnDelete rollout

Before its sync, require all OpenBao replicas initialized and unsealed. Save the original cluster ID and unseal/recovery material securely. On the active node, take and verify a Raft snapshot with bao operator raft snapshot save using the installation's approved authentication procedure. Copy it into the private recovery directory before replacing a pod.

After the target StatefulSet is applied, OnDelete pods may still run the old revision while the Application looks healthy. Replace one old-revision pod at a time, starting with the highest ordinal as in the release transition. Before each replacement, verify healthy quorum and unchanged unseal keys. Wait for that pod's target controller-revision-hash, Ready containers, initialized/unsealed status and unchanged cluster ID before continuing. Use the existing unseal procedure if needed; never run bao operator init on this existing cluster. For OnDelete, inspect pod revisions and updated replicas rather than assuming currentRevision == updateRevision is required.

LiteLLM PostgreSQL advisory lock

Prisma uses a session-level advisory lock. The old transaction pooler can leave it attached to an idle PostgreSQL session after the migrator finishes. Switching the new deployment to the session pooler prevents recurrence but does not release that old session automatically.

Inspect the lock on the CNPG primary without displaying credential values:

SELECT a.pid, a.backend_start, a.client_addr, a.state,
       a.query, a.xact_start, l.granted
FROM pg_locks l JOIN pg_stat_activity a ON a.pid = l.pid
WHERE l.locktype = 'advisory' AND l.classid = 0 AND l.objid = 72707369
  AND a.datname = 'litellm' AND a.usename = 'litellm';

If no matching lock exists, do not terminate sessions. For the classified stale lock, verify the client IP belongs to an existing CNPG transaction pooler pod, there is exactly one idle holder whose last query is COMMIT with no active transaction, and any waiters are only SELECT pg_advisory_lock(72707369). Stop the old LiteLLM workload under the reconciliation guard so it cannot reacquire the lock. A database operator may then terminate only those verified sessions, rechecking PID, backend start time, client IP and lock predicates immediately before each termination. Confirm the lock is gone before starting the target migrator. An active or unrecognized transaction requires diagnosis, not this cleanup.

Migrate administrator access in Keycloak

Platform administrators own this work; the upgrade CLI does not migrate user privileges. Technical upgrade completion does not mean that these assignments are finished. Plan any access interruption and keep a working Keycloak recovery login.

Migrate your administrator accounts explicitly. The new platform-administrators and tenant-administrators groups replace the legacy Portal administration model; they are not new names for the same groups. A user who could administer the 1.3.0 Portal through a legacy group does not automatically become a 1.4.0 administrator.

The standard 1.3.0 group names are COREAI-PORTAL-ADMIN-GROUP and COREAI-PORTAL-USERS-GROUP. Check the actual names, IDs, and memberships in your installation before making changes.

Choose the correct access scope

Platform administration does not make a user administrator of every customer tenant. Global is the specific inherited-administration case. Tenant administration does not grant platform authority, and administrator assignments are not a blanket replacement for resource sharing or ownership.

1. Inventory access before the identity cutover

Record the legacy groups' IDs, member user IDs, tenant memberships, and any explicit resource-sharing or role assignments that depend on those groups. Decide which former administrators need platform authority and which need tenant-only authority. Do not promote every legacy Portal administrator to platform administrator.

Keep a working Keycloak administrative recovery login and verify the configured bootstrap administrator account. The 1.4.0 platform bootstrap assigns only its configured administrator, identified by KEYCLOAK_ADMIN_USERNAME, to platform-administrators. It does not copy other legacy administrators. Newly provisioned tenant-administrators groups are empty until users are explicitly assigned.

For a manually staged identity cutover only: For operator-managed tenants, PlatformTenant.spec.identity.compatibilityGroups is the complete desired set of compatibility groups. An omitted or empty list allows previously managed legacy groups to be deleted during reconciliation. If the administrator-approved manual cutover requires those groups to remain temporarily, retain their exact existing names in this list before the upgraded operator reconciles them. This is a site-specific modification, not a step in the unchanged-baseline managed path. Preserve any other required entries. Do not put tenant-administrators in the list; it is managed separately.

2. Use the provisioned groups

As part of the coordinated identity, operator, API, and portal rollout, confirm that the canonical groups have been provisioned. The platform bootstrap owns the realm-level platform group and groups for bootstrap-declared Organizations; the tenant operator provisions the tenant group for operator-managed Organizations.

Do not rename a legacy group or manually create a lookalike. The backend validates group scope and management attributes, not just the displayed name:

• platform-administrators has bsqai.identity.managed-by=platform-bootstrap and bsqai.identity.role=platform.admin.
• Each tenant-administrators group has bsqai.identity.managed-by=platform-bootstrap, bsqai.identity.role=tenant.admin, and bsqai.identity.organization-id matching that tenant's Organization UUID. Operator-managed groups also carry their ownership metadata.

Preserve these attributes and do not attach Keycloak realm or client roles to the groups. Resolve missing groups or ownership conflicts through the owning bootstrap or tenant reconciliation before assigning users.

3. Assign the existing users

Use the Keycloak application realm configured for the Portal and BSQAI API, not just an administrator account in the master realm. Keep the existing user accounts and UUIDs.

1. For each approved platform administrator, add the user to the existing top-level platform-administrators group. Confirm the user is enabled and is also a member of the Global Organization. A direct group change in Keycloak does not itself add Global membership.
2. For each tenant administrator, open the target Organization, confirm the user is an enabled member, and add them to that Organization's existing tenant-administrators group.
3. Preserve ordinary tenant memberships. Replace any additional legacy access with the appropriate functional roles, custom roles, or resource grants rather than assuming an administrator assignment reproduces every former entitlement.

Once a platform administrator can use the upgraded Portal, prefer its dedicated Platform Administrators and Tenant Administrators controls for further assignments. Those flows manage the corresponding Keycloak memberships and apply administrator safeguards; the platform-administrator grant also establishes Global membership.

OIDC claim-to-group synchronization is not an administrator-migration mechanism. Canonical administrator groups are protected targets; use ordinary tenant groups for claim mappings and keep protected administrator assignments explicit.

4. Verify effective access before cleanup

Keycloak membership is projected into OpenFGA by the synchronization worker on a 30-second schedule. Wait for a successful synchronization cycle, then sign out and back in to refresh the Portal session. Signing in again does not replace a failed synchronization.

Verify with the migrated accounts:

• platform administrators can access platform administration and Global tenant settings;
• tenant-only administrators can manage their assigned tenants, but do not gain platform administration or unrelated tenants' settings;
• ordinary members retain their expected Chat & Work access without administrative controls;
• required model, data, and resource-sharing workflows still work with their intended grants.

If access is missing, check enabled user and Organization membership, group scope and attributes, and the Keycloak synchronization worker's errors. Do not work around missing application access by granting broad Keycloak realm-management roles or editing OpenFGA tuples directly.

5. Retire legacy groups after verification

Remove an operator-owned legacy group from compatibilityGroups only after its required memberships and dependent grants have been migrated. Reconciliation deletes that group. Unowned groups, including legacy groups outside the operator's ownership, are not automatically removed; review and retire them separately. Do not delete other service-access groups merely because their names also start with COREAI.

Retaining a legacy group preserves the identity for remaining references; it does not restore the removed 1.3.0 name-based permission mapping. Deleting a group loses its memberships, and recreating the same name produces a different group ID. Keep the migration inventory and backups: an image rollback does not restore those identities or grants.

Migrate ingestion to Data Hub

Airbyte retirement is destructive and assumes that the data owner has already extracted everything needed before the upgrade starts. The script does not import connections, schedules, credentials or checkpoints. It removes the bundled component and its dedicated databases, volumes, bucket, Ceph user, credentials and namespaces; these are not retained in the running platform.

1. Inventory the old sources, destinations, connector versions, schedules, stream selections, credentials and progress. Export required configuration/state and protect destination data under the recovery plan. If there is nothing to migrate, there is no connection-transfer task.
2. Prepare compatible source/destination images and connection definitions using the connector guide. Review write modes, cursors and primary keys. A similarly named connector is not proof of compatible checkpoints.
3. Agree the replacement ingestion boundary with the data owner. Validate the destination against the exported source state; use a separate prepared environment if transfer validation must precede retirement. Stop old schedules and avoid concurrent old/new writers unless explicitly supported.
4. Only after extraction and retirement prerequisites are complete, proceed with the platform upgrade. No separate machine-verifiable business-migration receipt is required by the CLI.
5. On the upgraded deployment, create the Data Hub connections and ingestions with the intended tenant permissions. Run a controlled transfer from the agreed boundary; verify counts, keys, duplicates and update behavior before enabling replacement schedules. Do not paste an Airbyte checkpoint into the new state store.

Do not retain old live resources indefinitely as part of the upgrade. Private exports/backups follow the agreed recovery retention policy and are distinct from the retired installation. Shared services and unrelated customer destination data must remain intact.

Upgrading 1.3.0 to 1.4.0: MLflow CPU limit

BullSequana AI 1.4.0 replaces the Bitnami MLflow chart with the official MLflow chart. Preserve the service account, databases, S3 data and credentials, and review the target chart values before syncing. Back up both mlflow and mlflow_auth; the new pod runs mlflow db upgrade in an init container, and reverting an image does not reverse its schema changes.

The platform adapts startup load to MLFLOW_RESOURCES_CPU_LIMIT. The minimum supported CPU limit is 500m:

Below 1 CPU, background job execution is disabled to reduce startup load. The chart has no startup probe, so slow startup can trigger liveness restarts. Do not assume a fixed readiness time or guaranteed uninterrupted service during migration. Explicit MLFLOW_SERVER_WORKERS and MLFLOW_SERVER_JOB_EXECUTION overrides must fit the actual CPU budget. Verify init completion, tracking-server health and existing artifact access before proceeding.

Verify retrieval and client compatibility

Before enabling Hybrid on existing knowledge collections, follow the BM25 migration sequence. Record the active collection and embedding configuration, preserve source files and cached chunks, and account for same-model recreation downtime. Do not delete an index while uploads or reindexing are still writing to it.

After reindexing, inspect completed, failed, and skipped file counts. Test semantic and exact-keyword queries, one newly ingested document, and a denied cross-tenant or private-file lookup. Existing embeddings clients must consume the indexed data[] response from /v1/embeddings; run the single/batch client example with a model available on the upgraded deployment.

Preserve tenant resource ownership

Upgrade the tenant-resource CRDs, RBAC, and controller together before adopting the new resource contracts. Back up operator ownership journals, canonical credentials, and provider data as well as tenant namespaces. For existing S3 tenants, verify bucket provenance and stage the scoped credential binding for the exact lifecycle and tenant-resource UID before allowing deletion.

Confirm MilvusReady or MLflowReady for the current generation before consuming a private binding. Test its authorized access and a denied foreign-tenant request. Schedule any credential-rotation test separately from the upgrade; preserve active credentials during this transition. The private MLflow authorizationReady gate is an installation prerequisite, not an automatic authorization test.

On ownership-recovery errors, restore the original protected journal and credentials; do not infer ownership from resource names or remove finalizers. Keep a controller that supports PlatformTenantResource while instances or cleanup finalizers exist. Reverting an operator or MLflow image does not reverse CRD changes or database migrations. See tenant-resource recovery.

Validate the upgraded deployment

Complete these checks for either path. Compare with the pre-upgrade evidence, not just an empty smoke-test account.

A successful technical upgrade does not assert unchanged effective permissions. Platform administrators complete the access changes described above and verify affected users before declaring their business workflows ready. Do not grant broader rights just to pass an upgrade test.

Related pages

• Platform CLI
• GitOps workflow
• Backup and disaster recovery
• BullSequana AI 1.4.0 release notes

Previous upgrade paths

• BullSequana AI 1.2.1 to 1.3.0: historical baseline, resumable CLI upgrade and manual migration procedure.