Upgrade 1.2.1 to 1.3.0

← Latest upgrade path

BullSequana AI 1.3.0 provides a customer-run, checkpointed upgrade for the classified 1.2.1 baseline. Deployments that differ from that baseline require remediation or a manually controlled migration.

Choose the right path

Do not use the fresh-deployment bootstrap sequence as an upgrade procedure.

Supported automated baseline

The versioned CLI accepts only 1.2.1 to 1.3.0. It does not infer other version jumps.

The supported baseline has:

• a platform root Application and the BullSequana AI app-of-apps hierarchy in namespace argocd
• Argo CD application v3.3.12, Helm release argo-cd, and chart 9.5.11
• cnpg-operator chart 0.28.3
• Keycloak chart 18.10.0
• Strimzi chart 0.50.1
• rook-ceph and rook-ceph-cluster charts v1.20.0
• no active Argo CD operation and no unclassified unhealthy or OutOfSync Application
• distinct AKS system and user node pools, CriticalAddonsOnly=true:NoSchedule on every system node, and at least one Ready user-pool node

The plan recognizes the missing 1.2.1 Airbyte migration Job after TTL cleanup as classified drift. It also tolerates the exact temporary Kubeflow source-normalization difference that Argo CD 10.x can report during the control-plane checkpoint. Any other drift fails the automated preflight.

When Azure DNS is enabled, AZURE_DNS_CLIENT_JSON must contain the same resourceGroup as AZURE_DNS_RESOURCE_GROUP_NAME. This prevents an apparently correct target render from retaining an embedded reference to an old cluster.

Before you start

Prepare the 1.3.0 platform checkout and install its CLI. The checkout's VERSION must contain 1.3.0.

uv sync
uv tool install --editable . --python 3.12

Use the target checkout's platform.env and local.env to define the 1.3.0 configuration. GIT_BRANCH must still identify the current 1.2.1 manifests branch; the upgrade creates a separate immutable ref from that baseline.

Place the active sealed-secrets material in the target checkout:

src/manifests/seal/sealed-secret.crt
src/manifests/seal/sealed-secret.key

Before changing the cluster:

• back up every service database and verify that the backup can be read
• export the current Argo CD Applications, AppProject, Helm release values, and deployed Git revision
• preserve the sealed-secrets private key and certificate in the approved secret store
• preserve platform.env, local.env, and generated secret values without printing credentials into logs
• confirm access to the manifests repository and the registry containing Argo CD chart 10.4.1
• identify custom manifests, component overrides, storage classes, routes, policies, and stateful-service changes
• define service-specific rollback criteria and owners

Never run apply-sealing-key during this upgrade. Replacing the sealing identity makes existing SealedSecret resources undecryptable.

Upgrade a standard 1.2.1 deployment

1. Run the read-only plan

bsqai upgrade plan --from 1.2.1 --to 1.3.0

The command prints the migration contract and a PASS or FAIL result for the live root Application, Application health, idle state, source version, chart fingerprint, node-pool isolation, Azure DNS target, and existing upgrade state. Resolve every failure before continuing.

upgrade plan --offline prints the contract without cluster access. It does not establish that a live deployment is upgradeable.

2. Validate the target render against live secrets

bsqai upgrade apply --from 1.2.1 --to 1.3.0 --dry-run

The live dry run:

• reads the classified 1.2.1 Secret data without printing it
• maps existing credentials into renamed 1.3.0 variables
• generates and persists only new 1.3.0 credentials in the ignored generated_secrets.env
• rejects an unclassified change to any Secret datum that exists in both versions
• reuses live SealedSecret ciphertext when the desired plaintext is unchanged
• renders every target Application against one candidate immutable ref
• performs no Git or Kubernetes mutation

Use --dry-run --offline only for structural rendering when cluster access is unavailable. Offline validation cannot prove credential continuity and is not a substitute for the live dry run.

3. Execute the upgrade

bsqai upgrade apply --from 1.2.1 --to 1.3.0

Do not start manual Argo CD syncs or change automated policies while the command runs. The default timeout is 3,600 seconds per Application operation. Use --timeout <seconds> when the approved maintenance window requires a different per-Application limit.

The command creates a ref named like upgrade/<platform-domain>/1.3.0-<UTC-timestamp>. A caller can instead supply one unused ref with --manifest-ref <ref>. Publication is create-only: the CLI refuses an existing remote ref.

4. Confirm completion

bsqai upgrade status

A completed state reports phase completed, the immutable ref and commit, and the number of completed component Applications. Run the post-upgrade validation before closing the maintenance window.

What the CLI changes

The upgrader performs these phases in order:

1. renders and publishes the immutable 1.3.0 snapshot;
2. stores a credential-free checkpoint in the argocd/bsqai-platform-upgrade ConfigMap;
3. preserves the Argo CD admin digest, Redis credential, and Keycloak client secret while using the configured Git and registry access to Helm-upgrade the existing control plane to Argo CD v3.5.2 through chart 10.4.1;
4. leaves the root Application on the 1.2.1 source until the upgraded control plane is stable;
5. adds a project-wide deny sync window, saves each Application's automated policy, and pauses automated reconciliation;
6. moves the root and all Git-backed parent and child sources to the immutable ref;
7. drives manual, pruned Argo CD operations in dependency and sync-wave order; and
8. restores the exact saved policies, requires full Synced and Healthy convergence with no failed hook Jobs, records platform version 1.3.0, and removes the temporary sync window last.

The parent order is common, inference, coreai, then proai. The Envoy AI Gateway CRDs and controller are bootstrapped from the new inference parent before the normal parent traversal so the upgraded Gateway data plane retains a working controller. Only explicitly catalogued, dependency-free Applications in the same wave run in parallel, with a maximum concurrency of three.

The migration also enforces these gates:

• the new inference parent takes ownership of model-serving resources before coreai can prune its former folder
• CNPG 1.30 installs the databaseroles.postgresql.cnpg.io CRD before DatabaseRole consumers run
• the Barman plugin has both TLS Secrets, a ready service endpoint, and confirmed CNPG operator registration before database clusters continue
• the Rook-Ceph CVE migration hook completes before the CephCluster changes
• the Keycloak hook waits for target CNPG readiness, creates and verifies a backup, and replaces only the incompatible StatefulSet while retaining the database and PVCs
• Keycloak Organization reconciliation finishes before dependent Applications
• Lakekeeper finishes before the Kubeflow profile-bucket provisioner runs
• the GitOps-native Kyverno migration Job runs with permanent least-scope RBAC
• the final gate checks live Application convergence and terminal Argo hook Jobs

Rook-Ceph security migration

This migration applies only when the platform manages object storage with Rook-Ceph. For an existing cluster, the PreSync preparation hook accepts the supported Ceph 20.2.1 source, verifies that the operator and storage data plane are healthy, and disables the rook manager module before the image update. The in-sync reconciler then moves Rook to 1.20.6 and Ceph to 20.2.4 and rotates Rook-managed daemon keys to CephX generation 2 with AES256K. CSI client keys remain AES in this release for compatibility with older kernels. The migration records its completion and is safe to reconcile again.

Do not continue past the storage gate until all of the following are true:

• Rook reports 1.20.6, every Ceph daemon reports 20.2.4, and the cluster is healthy
• the CephCluster and object-store status.cephx fields report generation 2 for the Rook-managed daemon keys
• AUTH_INSECURE_SERVICE_KEY_TYPE and AUTH_INSECURE_SERVICE_TICKETS are absent; only AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE, AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED, and AUTH_INSECURE_KEYS_CREATABLE may be muted for the client transition
• existing RBD and CephFS workloads pass read/write checks, and RGW passes a write/read/delete smoke test
• rook-ceph-security-migrate-v20-2-4 is complete and its migration state records phase complete

A one-replica RGW endpoint can be briefly unavailable while its pod rolls. A site that requires uninterrupted S3 access must provide at least two RGW replicas with topology spread or anti-affinity, an appropriate PodDisruptionBudget, and a continuous-I/O probe during the upgrade. The migration Job cannot compensate for a single-replica serving topology.

Resume a stopped upgrade

When an operation fails, the CLI records phase failed and the last error. It deliberately leaves the project sync window and paused policies in place.

bsqai upgrade status

Correct the reported runtime, dependency, or capacity problem, then rerun the same command:

bsqai upgrade apply --from 1.2.1 --to 1.3.0

The CLI reuses the same manifest ref and commit, skips completed Applications, and resumes the interrupted parent operation. Do not specify a different --manifest-ref, delete the checkpoint ConfigMap, remove the sync window, or restore automation manually.

If the failure is caused by an error in the rendered target manifests, correct and validate the 1.3.0 source, then publish a replacement snapshot through the explicit recovery option:

bsqai upgrade apply --from 1.2.1 --to 1.3.0 --supersede-failed-snapshot

This option is accepted only for a failed checkpoint and cannot be combined with --manifest-ref. It creates another immutable ref, replays the root and parent handoff under the existing lock, retains completed component checkpoints, and resyncs completed Applications only when their desired resources changed.

The upgrade command is resumable; it is not a general rollback engine. Do not switch the root ref back after a stateful migration and assume the data plane has rolled back. Use the recorded backups and the service-specific recovery plan when rollback criteria are met.

Manual migration for advanced deployments

Use this path when the live environment does not match the supported fingerprint or when site changes make the automated ordering unsafe. The deployment owner is responsible for adapting every step to the recorded differences and proving credential, storage, identity, and data continuity.

1. Classify the local differences

Run the read-only plan and retain its output even when failure is expected:

bsqai upgrade plan --from 1.2.1 --to 1.3.0

For every failed check and local modification, record:

• the live Application, chart version, Git source, resource, or field that differs
• whether the 1.3.0 target keeps, replaces, migrates, or removes it
• dependencies that must be ready before it changes
• the data, credential, and ownership boundary it affects
• the validation and rollback action

Do not present a modified environment as the standard baseline by changing only the version annotation or CLI arguments.

2. Protect state and credentials

Complete the backups and exports in Before you start. Map every existing credential into the 1.3.0 configuration. New target-only credentials must be generated once and stored through the site's approved secret process.

Run the live dry run when the site's Secret names and namespaces still match the classified contract:

bsqai upgrade apply --from 1.2.1 --to 1.3.0 --dry-run

If local Secret names or data keys differ, reproduce the same continuity check manually: unseal the target output with the existing recovery key, compare overlapping data byte for byte, approve only documented 1.3.0 transitions, and reseal with the existing certificate. Never commit plaintext Secret data.

3. Create one immutable target snapshot

Choose a unique ref and render every Application against it:

export BASELINE_GIT_BRANCH="<current-1.2.1-manifests-branch>"
export UPGRADE_REF="upgrade/<site-domain>/1.3.0-<UTC-timestamp>"
export GIT_BRANCH="$BASELINE_GIT_BRANCH"

bsqai upgrade apply \
  --from 1.2.1 \
  --to 1.3.0 \
  --manifest-ref "$UPGRADE_REF" \
  --dry-run

If custom Secret names prevent the classified live import, populate the target credential variables from the approved secret store and add --offline. Then perform the unseal-and-compare continuity review described in the previous step; the offline render does not perform it for you.

Review src/manifests/output/. Every non-chart source in every child Application and every parent source must use $UPGRADE_REF.

Publish the reviewed output once:

export GIT_BRANCH="$UPGRADE_REF"
bsqai push-manifests "Manual BullSequana AI 1.2.1 to 1.3.0 snapshot"
export GIT_BRANCH="$BASELINE_GIT_BRANCH"

For a manual migration, push-manifests does not enforce create-only publication. Verify that the ref does not exist before the push, protect it against updates afterward, and record its commit hash.

4. Upgrade the existing Argo CD control plane

Keep the platform root Application on the 1.2.1 ref. Prepare reviewed Argo CD values that preserve the live admin password digest, Redis password, Keycloak client secret, Git credential, and registry credential. Do not use fresh bootstrap values.

Upgrade the existing release rather than installing a second control plane:

helm upgrade argo-cd \
  oci://<registry>/<helm-repository>/argo-cd \
  --namespace argocd \
  --version 10.4.1 \
  --values <reviewed-argocd-values.yaml> \
  --wait \
  --timeout <site-timeout>

Confirm that Argo CD v3.5.2, the root Application, repository access, SSO, and all baseline Applications are stable before changing Git sources.

5. Lock automated reconciliation

Append this deny window to the existing default AppProject without discarding site-defined windows:

kind: deny
schedule: "* * * * *"
duration: 24h
applications:
  - "*"
manualSync: true
timeZone: UTC

Export every Application's spec.syncPolicy.automated value, then remove that field from all existing Applications. Repeat this capture-and-pause step for parent or child Applications created during the cutover.

Manual syncs must prune, preserve each Application's declared sync options, and use ServerSideApply=true when no SSA setting is declared. App-of-apps operations that ignore /spec/syncPolicy/automated must also use RespectIgnoreDifferences=true so a parent does not restore a paused child prematurely.

6. Move the hierarchy to the snapshot

Set the platform Application's Git targetRevision to $UPGRADE_REF and manually sync it. Confirm that each enabled parent appears and references the same ref, then pause automation on the new parents.

Before normal parent traversal, sync envoy-ai-gateway-crds and envoy-ai-gateway from the inference parent. Confirm that service envoy-ai-gateway-system/ai-gateway-controller has a ready endpoint on port 1063.

Continue one parent at a time in this order:

1. common
2. inference
3. coreai
4. proai

For each parent, update it to $UPGRADE_REF, start one manual parent operation, verify that every child Git source uses the same ref, and sync children by their declared wave. Do not allow coreai to prune its old inference folder until the new inference parent and its workloads are healthy.

7. Enforce the migration gates

Use this order where the listed Applications are enabled:

Skip an Application that is intentionally disabled. Do not create a component solely to match this table; document the equivalent site-specific dependency instead.

8. Recreate only classified disposable resources

Delete each applicable resource immediately before syncing its owning Application. Use foreground deletion and wait for absence so Argo CD can recreate it from the snapshot.

Confirm that the Speaches PVC contains only reproducible model-cache data before deletion. Do not generalize this recreation list to other PVCs, databases, StatefulSets, or Secrets. The Keycloak PreSync hook owns its separately gated StatefulSet replacement.

9. Restore automation last

While the project-wide deny window remains active:

1. finish every parent operation;
2. reconcile any changed child still OutOfSync;
3. restore each exact saved spec.syncPolicy.automated value;
4. require every enabled Application to be Synced and Healthy;
5. inspect tracked hook Jobs and resolve any terminal failure;
6. annotate the platform Application with bsqai.ai/platform-version: 1.3.0; and
7. remove only the temporary deny window, preserving all prior AppProject windows.

If the manual migration stops, keep the lock and paused policies in place. Correct the cause and continue from the last verified gate. A rollback after a database, storage, or identity migration must use that service's recorded recovery plan and backups rather than only changing Git refs.

Post-upgrade validation

Confirm:

• all four expected parent Applications are present when enabled and every Git source uses the recorded immutable ref
• all enabled Applications are Synced and Healthy and no tracked hook Job remains failed
• the platform Application records version 1.3.0
• existing sealed secrets decrypt without credential rotation
• authentication and tenant selection work for single- and multi-Organization users
• tenant creation and deletion reconcile identity and selected optional resources
• KServe models deploy, report health, and answer through the BSQAI API
• object-storage reads, writes, and browser-facing presigned URLs work
• CNPG backups continue through the registered Barman plugin and a restore check succeeds
• JSON pod logs carrying a top-level tenant_id reach that tenant's Loki partition, while unscoped and OTLP logs reach platform
• portal, API, MCP, Files and RAG, and Data smoke tests succeed

Release engineering

Platform releases follow semantic versioning from the repository VERSION file. Deployment automation must select an immutable release branch or tag and matching portal, backend, image, and chart artifacts.

Treat image-scan output, integration-test results, release notes, the immutable manifest commit, backup evidence, and the completed upgrade validation record as release evidence.

Related pages

• Platform CLI
• Prerequisites
• GitOps workflow
• Backup and disaster recovery
• BullSequana AI 1.3.0 release notes