# Upgrade 1.2.1 to 1.3.0 (/docs/deployment/playbooks/upgrade-1-2-1-to-1-3-0) [← Latest upgrade path](/docs/deployment/playbooks/upgrade-and-release-process) BullSequana AI 1.3.0 provides a customer-run, checkpointed upgrade for the classified 1.2.1 baseline. Deployments that differ from that baseline require remediation or a manually controlled migration. ## Choose the right path [#choose-the-right-path] | Starting point | Path | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Fresh deployment | Follow the [deployment sequence](/docs/deployment/playbooks/deployment-sequence). Bootstrap a new sealing key and Argo CD only for this path. | | Standard BullSequana AI 1.2.1 deployment on AKS | Run the `upgrade plan`, live dry run, and resumable `upgrade apply` procedure on this page. | | Modified 1.2.1, alternate Argo CD topology, or non-AKS deployment | Run the plan to identify differences, then restore the supported baseline or follow [Manual migration for advanced deployments](#manual-migration-for-advanced-deployments). | Do not use the fresh-deployment bootstrap sequence as an upgrade procedure. ## Supported automated baseline [#supported-automated-baseline] The versioned CLI accepts only `1.2.1` to `1.3.0`. It does not infer other version jumps. The supported baseline has: * a `platform` root Application and the BullSequana AI app-of-apps hierarchy in namespace `argocd` * Argo CD application `v3.3.12`, Helm release `argo-cd`, and chart `9.5.11` * `cnpg-operator` chart `0.28.3` * Keycloak chart `18.10.0` * Strimzi chart `0.50.1` * `rook-ceph` and `rook-ceph-cluster` charts `v1.20.0` * no active Argo CD operation and no unclassified unhealthy or OutOfSync Application * distinct AKS system and user node pools, `CriticalAddonsOnly=true:NoSchedule` on every system node, and at least one Ready user-pool node The plan recognizes the missing 1.2.1 Airbyte migration Job after TTL cleanup as classified drift. It also tolerates the exact temporary Kubeflow source-normalization difference that Argo CD 10.x can report during the control-plane checkpoint. Any other drift fails the automated preflight. When Azure DNS is enabled, `AZURE_DNS_CLIENT_JSON` must contain the same `resourceGroup` as `AZURE_DNS_RESOURCE_GROUP_NAME`. This prevents an apparently correct target render from retaining an embedded reference to an old cluster. ## Before you start [#before-you-start] Prepare the 1.3.0 platform checkout and install its CLI. The checkout's `VERSION` must contain `1.3.0`. ```bash uv sync uv tool install --editable . --python 3.12 ``` Use the target checkout's `platform.env` and `local.env` to define the 1.3.0 configuration. `GIT_BRANCH` must still identify the current 1.2.1 manifests branch; the upgrade creates a separate immutable ref from that baseline. Place the active sealed-secrets material in the target checkout: ```text src/manifests/seal/sealed-secret.crt src/manifests/seal/sealed-secret.key ``` Before changing the cluster: * back up every service database and verify that the backup can be read * export the current Argo CD Applications, AppProject, Helm release values, and deployed Git revision * preserve the sealed-secrets private key and certificate in the approved secret store * preserve `platform.env`, `local.env`, and generated secret values without printing credentials into logs * confirm access to the manifests repository and the registry containing Argo CD chart `10.4.1` * identify custom manifests, component overrides, storage classes, routes, policies, and stateful-service changes * define service-specific rollback criteria and owners Never run `apply-sealing-key` during this upgrade. Replacing the sealing identity makes existing `SealedSecret` resources undecryptable. ## Upgrade a standard 1.2.1 deployment [#upgrade-a-standard-121-deployment] ### 1. Run the read-only plan [#1-run-the-read-only-plan] ```bash bsqai upgrade plan --from 1.2.1 --to 1.3.0 ``` The command prints the migration contract and a PASS or FAIL result for the live root Application, Application health, idle state, source version, chart fingerprint, node-pool isolation, Azure DNS target, and existing upgrade state. Resolve every failure before continuing. `upgrade plan --offline` prints the contract without cluster access. It does not establish that a live deployment is upgradeable. ### 2. Validate the target render against live secrets [#2-validate-the-target-render-against-live-secrets] ```bash bsqai upgrade apply --from 1.2.1 --to 1.3.0 --dry-run ``` The live dry run: * reads the classified 1.2.1 Secret data without printing it * maps existing credentials into renamed 1.3.0 variables * generates and persists only new 1.3.0 credentials in the ignored `generated_secrets.env` * rejects an unclassified change to any Secret datum that exists in both versions * reuses live SealedSecret ciphertext when the desired plaintext is unchanged * renders every target Application against one candidate immutable ref * performs no Git or Kubernetes mutation Use `--dry-run --offline` only for structural rendering when cluster access is unavailable. Offline validation cannot prove credential continuity and is not a substitute for the live dry run. ### 3. Execute the upgrade [#3-execute-the-upgrade] ```bash bsqai upgrade apply --from 1.2.1 --to 1.3.0 ``` Do not start manual Argo CD syncs or change automated policies while the command runs. The default timeout is 3,600 seconds per Application operation. Use `--timeout ` when the approved maintenance window requires a different per-Application limit. The command creates a ref named like `upgrade//1.3.0-`. A caller can instead supply one unused ref with `--manifest-ref `. Publication is create-only: the CLI refuses an existing remote ref. ### 4. Confirm completion [#4-confirm-completion] ```bash bsqai upgrade status ``` A completed state reports phase `completed`, the immutable ref and commit, and the number of completed component Applications. Run the [post-upgrade validation](#post-upgrade-validation) before closing the maintenance window. ## What the CLI changes [#what-the-cli-changes] The upgrader performs these phases in order: 1. renders and publishes the immutable 1.3.0 snapshot; 2. stores a credential-free checkpoint in the `argocd/bsqai-platform-upgrade` ConfigMap; 3. preserves the Argo CD admin digest, Redis credential, and Keycloak client secret while using the configured Git and registry access to Helm-upgrade the existing control plane to Argo CD `v3.5.2` through chart `10.4.1`; 4. leaves the root Application on the 1.2.1 source until the upgraded control plane is stable; 5. adds a project-wide deny sync window, saves each Application's automated policy, and pauses automated reconciliation; 6. moves the root and all Git-backed parent and child sources to the immutable ref; 7. drives manual, pruned Argo CD operations in dependency and sync-wave order; and 8. restores the exact saved policies, requires full Synced and Healthy convergence with no failed hook Jobs, records platform version `1.3.0`, and removes the temporary sync window last. The parent order is `common`, `inference`, `coreai`, then `proai`. The Envoy AI Gateway CRDs and controller are bootstrapped from the new `inference` parent before the normal parent traversal so the upgraded Gateway data plane retains a working controller. Only explicitly catalogued, dependency-free Applications in the same wave run in parallel, with a maximum concurrency of three. The migration also enforces these gates: * the new `inference` parent takes ownership of model-serving resources before `coreai` can prune its former folder * CNPG 1.30 installs the `databaseroles.postgresql.cnpg.io` CRD before DatabaseRole consumers run * the Barman plugin has both TLS Secrets, a ready service endpoint, and confirmed CNPG operator registration before database clusters continue * the Rook-Ceph CVE migration hook completes before the CephCluster changes * the Keycloak hook waits for target CNPG readiness, creates and verifies a backup, and replaces only the incompatible StatefulSet while retaining the database and PVCs * Keycloak Organization reconciliation finishes before dependent Applications * Lakekeeper finishes before the Kubeflow profile-bucket provisioner runs * the GitOps-native Kyverno migration Job runs with permanent least-scope RBAC * the final gate checks live Application convergence and terminal Argo hook Jobs ## Rook-Ceph security migration [#rook-ceph-security-migration] This migration applies only when the platform manages object storage with Rook-Ceph. For an existing cluster, the PreSync preparation hook accepts the supported Ceph 20.2.1 source, verifies that the operator and storage data plane are healthy, and disables the `rook` manager module before the image update. The in-sync reconciler then moves Rook to 1.20.6 and Ceph to 20.2.4 and rotates Rook-managed daemon keys to CephX generation 2 with AES256K. CSI client keys remain AES in this release for compatibility with older kernels. The migration records its completion and is safe to reconcile again. Do not continue past the storage gate until all of the following are true: * Rook reports 1.20.6, every Ceph daemon reports 20.2.4, and the cluster is healthy * the CephCluster and object-store `status.cephx` fields report generation 2 for the Rook-managed daemon keys * `AUTH_INSECURE_SERVICE_KEY_TYPE` and `AUTH_INSECURE_SERVICE_TICKETS` are absent; only `AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE`, `AUTH_INSECURE_CLIENT_KEY_TYPE`, `AUTH_INSECURE_KEYS_ALLOWED`, and `AUTH_INSECURE_KEYS_CREATABLE` may be muted for the client transition * existing RBD and CephFS workloads pass read/write checks, and RGW passes a write/read/delete smoke test * `rook-ceph-security-migrate-v20-2-4` is complete and its migration state records phase `complete` A one-replica RGW endpoint can be briefly unavailable while its pod rolls. A site that requires uninterrupted S3 access must provide at least two RGW replicas with topology spread or anti-affinity, an appropriate PodDisruptionBudget, and a continuous-I/O probe during the upgrade. The migration Job cannot compensate for a single-replica serving topology. ## Resume a stopped upgrade [#resume-a-stopped-upgrade] When an operation fails, the CLI records phase `failed` and the last error. It deliberately leaves the project sync window and paused policies in place. ```bash bsqai upgrade status ``` Correct the reported runtime, dependency, or capacity problem, then rerun the same command: ```bash bsqai upgrade apply --from 1.2.1 --to 1.3.0 ``` The CLI reuses the same manifest ref and commit, skips completed Applications, and resumes the interrupted parent operation. Do not specify a different `--manifest-ref`, delete the checkpoint ConfigMap, remove the sync window, or restore automation manually. If the failure is caused by an error in the rendered target manifests, correct and validate the 1.3.0 source, then publish a replacement snapshot through the explicit recovery option: ```bash bsqai upgrade apply --from 1.2.1 --to 1.3.0 --supersede-failed-snapshot ``` This option is accepted only for a failed checkpoint and cannot be combined with `--manifest-ref`. It creates another immutable ref, replays the root and parent handoff under the existing lock, retains completed component checkpoints, and resyncs completed Applications only when their desired resources changed. The upgrade command is resumable; it is not a general rollback engine. Do not switch the root ref back after a stateful migration and assume the data plane has rolled back. Use the recorded backups and the service-specific recovery plan when rollback criteria are met. ## Manual migration for advanced deployments [#manual-migration-for-advanced-deployments] Use this path when the live environment does not match the supported fingerprint or when site changes make the automated ordering unsafe. The deployment owner is responsible for adapting every step to the recorded differences and proving credential, storage, identity, and data continuity. During the controlled cutover, either the upgrade operator or Argo CD automation owns reconciliation. Do not leave automated sync enabled while also applying the steps manually. ### 1. Classify the local differences [#1-classify-the-local-differences] Run the read-only plan and retain its output even when failure is expected: ```bash bsqai upgrade plan --from 1.2.1 --to 1.3.0 ``` For every failed check and local modification, record: * the live Application, chart version, Git source, resource, or field that differs * whether the 1.3.0 target keeps, replaces, migrates, or removes it * dependencies that must be ready before it changes * the data, credential, and ownership boundary it affects * the validation and rollback action Do not present a modified environment as the standard baseline by changing only the version annotation or CLI arguments. ### 2. Protect state and credentials [#2-protect-state-and-credentials] Complete the backups and exports in [Before you start](#before-you-start). Map every existing credential into the 1.3.0 configuration. New target-only credentials must be generated once and stored through the site's approved secret process. Run the live dry run when the site's Secret names and namespaces still match the classified contract: ```bash bsqai upgrade apply --from 1.2.1 --to 1.3.0 --dry-run ``` If local Secret names or data keys differ, reproduce the same continuity check manually: unseal the target output with the existing recovery key, compare overlapping data byte for byte, approve only documented 1.3.0 transitions, and reseal with the existing certificate. Never commit plaintext Secret data. ### 3. Create one immutable target snapshot [#3-create-one-immutable-target-snapshot] Choose a unique ref and render every Application against it: ```bash export BASELINE_GIT_BRANCH="" export UPGRADE_REF="upgrade//1.3.0-" export GIT_BRANCH="$BASELINE_GIT_BRANCH" bsqai upgrade apply \ --from 1.2.1 \ --to 1.3.0 \ --manifest-ref "$UPGRADE_REF" \ --dry-run ``` If custom Secret names prevent the classified live import, populate the target credential variables from the approved secret store and add `--offline`. Then perform the unseal-and-compare continuity review described in the previous step; the offline render does not perform it for you. Review `src/manifests/output/`. Every non-chart source in every child Application and every parent source must use `$UPGRADE_REF`. Publish the reviewed output once: ```bash export GIT_BRANCH="$UPGRADE_REF" bsqai push-manifests "Manual BullSequana AI 1.2.1 to 1.3.0 snapshot" export GIT_BRANCH="$BASELINE_GIT_BRANCH" ``` For a manual migration, `push-manifests` does not enforce create-only publication. Verify that the ref does not exist before the push, protect it against updates afterward, and record its commit hash. ### 4. Upgrade the existing Argo CD control plane [#4-upgrade-the-existing-argo-cd-control-plane] Keep the `platform` root Application on the 1.2.1 ref. Prepare reviewed Argo CD values that preserve the live admin password digest, Redis password, Keycloak client secret, Git credential, and registry credential. Do not use fresh bootstrap values. Upgrade the existing release rather than installing a second control plane: ```bash helm upgrade argo-cd \ oci:////argo-cd \ --namespace argocd \ --version 10.4.1 \ --values \ --wait \ --timeout ``` Confirm that Argo CD `v3.5.2`, the root Application, repository access, SSO, and all baseline Applications are stable before changing Git sources. ### 5. Lock automated reconciliation [#5-lock-automated-reconciliation] Append this deny window to the existing `default` AppProject without discarding site-defined windows: ```yaml kind: deny schedule: "* * * * *" duration: 24h applications: - "*" manualSync: true timeZone: UTC ``` Export every Application's `spec.syncPolicy.automated` value, then remove that field from all existing Applications. Repeat this capture-and-pause step for parent or child Applications created during the cutover. Manual syncs must prune, preserve each Application's declared sync options, and use `ServerSideApply=true` when no SSA setting is declared. App-of-apps operations that ignore `/spec/syncPolicy/automated` must also use `RespectIgnoreDifferences=true` so a parent does not restore a paused child prematurely. ### 6. Move the hierarchy to the snapshot [#6-move-the-hierarchy-to-the-snapshot] Set the `platform` Application's Git `targetRevision` to `$UPGRADE_REF` and manually sync it. Confirm that each enabled parent appears and references the same ref, then pause automation on the new parents. Before normal parent traversal, sync `envoy-ai-gateway-crds` and `envoy-ai-gateway` from the `inference` parent. Confirm that service `envoy-ai-gateway-system/ai-gateway-controller` has a ready endpoint on port `1063`. Continue one parent at a time in this order: 1. `common` 2. `inference` 3. `coreai` 4. `proai` For each parent, update it to `$UPGRADE_REF`, start one manual parent operation, verify that every child Git source uses the same ref, and sync children by their declared wave. Do not allow `coreai` to prune its old inference folder until the new `inference` parent and its workloads are healthy. ### 7. Enforce the migration gates [#7-enforce-the-migration-gates] Use this order where the listed Applications are enabled: | Order | Application or gate | Required evidence before continuing | | ----: | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 1 | Envoy AI Gateway bootstrap | `envoy-ai-gateway-crds` and `envoy-ai-gateway` are healthy and the controller endpoint is ready before the `common` parent changes Gateway API | | 2 | `sealed-secrets`, `cert-manager`, `trust-manager`, then `gateway-api` in `common` | Existing secrets decrypt, certificate controllers are healthy, and the generated Envoy Gateway data-plane Deployment finishes its rollout | | 3 | `cnpg-operator` | `databaseroles.postgresql.cnpg.io` exists | | 4 | `cnpg-barman-plugin` | Both Barman TLS Secrets and service port `9090` are ready; restart the CNPG operator pod and confirm registration of `barman-cloud.cloudnative-pg.io` in its logs | | 5 | `rook-ceph`, `ceph-csi-drivers`, `rook-ceph-cluster` | The CVE migration hook succeeds; Ceph reaches the target version and healthy state before dependent storage users continue | | 6 | `cnpg`, `keycloak`, then remaining `common` Applications by wave | Database clusters are healthy; the Keycloak PreSync backup is verified, the target StatefulSet is healthy, and Organization reconciliation completes | | 7 | Remaining `inference` Applications | Gateway API Inference Extension CRDs, KServe CRDs, KAI Scheduler, internal gateway, runtime configs, and KServe resources converge before model traffic is tested | | 8 | `coreai` Applications | Model serving is already owned by `inference`; API, portal, workflow, retrieval, and MCP services converge by wave | | 9 | `proai` Applications, including `strimzi` and `lakekeeper` before `kubeflow` | Streaming and Data services are healthy, Lakekeeper's application principal exists before the profile-bucket provisioner runs, and Kyverno's migration hook completes without a failed Job | Skip an Application that is intentionally disabled. Do not create a component solely to match this table; document the equivalent site-specific dependency instead. ### 8. Recreate only classified disposable resources [#8-recreate-only-classified-disposable-resources] Delete each applicable resource immediately before syncing its owning Application. Use foreground deletion and wait for absence so Argo CD can recreate it from the snapshot. | Application | Resource | Namespace/name | Reason | | ------------- | ----------------------- | -------------------------------------------- | ------------------------------------------------------------------ | | `temporal` | `Job` | `temporal/temporal-schema-1` | The chart changes the completed schema Job pod template | | `argo-events` | `Deployment` | `argo-events/argo-events-controller-manager` | The chart changes the Deployment strategy shape | | `pgadmin4` | `Deployment` | `pgadmin4/pgadmin4` | The strategy changes from RollingUpdate to Recreate | | `superset` | `Deployment` | `superset/superset` | The upgraded chart changes immutable selector labels | | `superset` | `Deployment` | `superset/superset-worker` | The upgraded chart changes immutable selector labels | | `llm-backend` | `Deployment` | `llm-backend/speaches` | The speech service changes strategy and cache-volume contract | | `llm-backend` | `Job` | `llm-backend/speaches-model-cache` | The completed cache Job retains the disposable cache PVC finalizer | | `llm-backend` | `PersistentVolumeClaim` | `llm-backend/speaches-model-cache` | The disposable model-cache PVC fields are immutable | Confirm that the Speaches PVC contains only reproducible model-cache data before deletion. Do not generalize this recreation list to other PVCs, databases, StatefulSets, or Secrets. The Keycloak PreSync hook owns its separately gated StatefulSet replacement. ### 9. Restore automation last [#9-restore-automation-last] While the project-wide deny window remains active: 1. finish every parent operation; 2. reconcile any changed child still OutOfSync; 3. restore each exact saved `spec.syncPolicy.automated` value; 4. require every enabled Application to be Synced and Healthy; 5. inspect tracked hook Jobs and resolve any terminal failure; 6. annotate the `platform` Application with `bsqai.ai/platform-version: 1.3.0`; and 7. remove only the temporary deny window, preserving all prior AppProject windows. If the manual migration stops, keep the lock and paused policies in place. Correct the cause and continue from the last verified gate. A rollback after a database, storage, or identity migration must use that service's recorded recovery plan and backups rather than only changing Git refs. ## Post-upgrade validation [#post-upgrade-validation] Confirm: * all four expected parent Applications are present when enabled and every Git source uses the recorded immutable ref * all enabled Applications are Synced and Healthy and no tracked hook Job remains failed * the `platform` Application records version `1.3.0` * existing sealed secrets decrypt without credential rotation * authentication and tenant selection work for single- and multi-Organization users * tenant creation and deletion reconcile identity and selected optional resources * KServe models deploy, report health, and answer through the BSQAI API * object-storage reads, writes, and browser-facing presigned URLs work * CNPG backups continue through the registered Barman plugin and a restore check succeeds * JSON pod logs carrying a top-level `tenant_id` reach that tenant's Loki partition, while unscoped and OTLP logs reach `platform` * portal, API, MCP, Files and RAG, and Data smoke tests succeed ## Release engineering [#release-engineering] Platform releases follow semantic versioning from the repository `VERSION` file. Deployment automation must select an immutable release branch or tag and matching portal, backend, image, and chart artifacts. Treat image-scan output, integration-test results, release notes, the immutable manifest commit, backup evidence, and the completed upgrade validation record as release evidence. ## Related pages [#related-pages] * [Platform CLI](/docs/deployment/platform-cli) * [Prerequisites](/docs/deployment/prerequirements) * [GitOps workflow](/docs/deployment/gitops-workflow) * [Backup and disaster recovery](/docs/deployment/playbooks/backup-and-disaster-recovery) * [BullSequana AI 1.3.0 release notes](/docs/release-notes/1-3-0)