Upgrade Temporal to 1.32
Check, resume, or run by hand the Temporal 1.29 to 1.32 migration of the BullSequana AI 2.0.0 upgrade, and restore Temporal from a backup.
BullSequana AI 1.4.0 runs Temporal server 1.29 with Elasticsearch for workflow search. BullSequana AI 2.0.0 runs Temporal 1.32 with workflow search in PostgreSQL. Temporal upgrades one minor release at a time, so 2.0.0 runs 1.30 and 1.31 against your database before it deploys 1.32.
By default the temporal Argo CD Application does this itself. Its PreSync Job temporal-migration checks the running Temporal and then does one of three things:
- On a fresh install, a cluster already on 1.31 or later, or a re-sync, it does nothing.
- On a standard 1.4.0 Temporal, it stops the 1.29 server and runs 1.30.7, then 1.31.3. Each release updates both schemas and then runs as one process until it is healthy. The chart then deploys 1.32.0.
- On any other state, it changes nothing and fails the sync. Temporal 1.4.0 keeps serving.
Temporal is unavailable for a few minutes during the migration. Workflow state is kept. Workflow search history from 1.4.0 is not carried over: Elasticsearch kept it for three days, and the 2.0.0 sync removes Elasticsearch and its volumes.
The commands below use the default names: namespace temporal, release temporal, and databases temporal and temporal_visibility in the CNPG cluster cnpg-cluster. Replace them if your installation overrides them.
The managed upgrade backs up the Temporal databases before the temporal sync and checks that the migration completed. On the manual GitOps upgrade, take the backup below yourself. Do not push the 2.0.0 manifests onto the branch the cluster syncs from while automated sync is on: the temporal application could then sync the new values with the old chart before its parent application updates the chart version. The manual upgrade holds reconciliation with a deny sync window first.
Back up the Temporal databases first
On the manual path, take this backup before you sync 2.0.0.
PRIMARY=$(kubectl -n cnpg get cluster cnpg-cluster -o jsonpath='{.status.currentPrimary}')
for db in temporal temporal_visibility; do
kubectl -n cnpg exec "$PRIMARY" -c postgres -- pg_dump -U postgres -d "$db" > "temporal-backup-$db.sql"
grep -q "PostgreSQL database dump complete" "temporal-backup-$db.sql" || echo "INCOMPLETE: $db"
doneKeep both files until 2.0.0 runs correctly. In 1.4.0 the temporal_visibility database is empty, because search ran in Elasticsearch.
The 2.0.0 sync deletes the Elasticsearch volume that held workflow search. If you want to keep that search history, back up the PersistentVolumeClaim elasticsearch-master-elasticsearch-master-0 in the temporal namespace with your storage tooling before you sync. Temporal 2.0.0 does not read it back.
Check the migration
The Job records its progress in the ConfigMap temporal-migration-status:
kubectl -n temporal get configmap temporal-migration-status -o jsonpath='{.data}'| Field | Meaning |
|---|---|
phase | in-progress while releases are running, done after the last one |
from | the server image tag found before the migration, for example 1.29.6.1 |
steps | the releases that finished, in order, for example 1.30.7,1.31.3 |
to | the last release the migration ran |
started, updated, finished | UTC timestamps |
Read what the Job decided and did:
kubectl -n temporal logs job/temporal-migration -c plan
kubectl -n temporal logs job/temporal-migration -c step-1-30-7
kubectl -n temporal logs job/temporal-migration -c step-1-31-3After the sync, the four server Deployments run 1.32.0, and the PostSync Job temporal-elasticsearch-cleanup has deleted the Elasticsearch volumes:
kubectl -n temporal get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image
kubectl -n temporal get pvc -l app=elasticsearch-masterResume a partial migration
A migration that stops partway leaves phase: in-progress. The old server Deployments stay at zero replicas, so Temporal is down until the migration finishes.
-
Find the release that failed. The failed container is the last
step-*container with a non-zero exit code.kubectl -n temporal get pods -l job-name=temporal-migration \ -o jsonpath='{range .items[*].status.initContainerStatuses[*]}{.name}={.state.terminated.exitCode}{"\n"}{end}' -
Read that container's log. On a failure it prints the last lines of the server log.
-
Fix the cause. A database that does not accept connections is the usual cause. Check the CNPG cluster and the Secret
temporal-postgresql-secret. -
Sync the
temporalApplication again. The new Job readssteps, skips the releases that finished, and runs the rest. A release that stopped halfway runs again from its schema update. The schema tools apply only the missing changes.
Never set the Deployments back to 1.29 after a release has run. Temporal 1.29 cannot run on a schema that 1.30 or 1.31 updated. To go back to 1.4.0, restore the backup.
After the migration stopped without changing anything
The sync fails with Temporal migration stopped before changing anything: and a reason. Temporal 1.4.0 keeps serving, and the ConfigMap is not created. Argo CD retries the sync, and the Job stops again until the state matches what it recognises.
| Reason in the message | What to do |
|---|---|
the Temporal server Deployments are incomplete or run different image tags | Finish or roll back the change on the Deployments, so that temporal-frontend, -history, -matching and -worker all exist and run one tag. Then sync again. |
the Temporal server image tag ... is not a release version | The Deployments run a custom tag or digest. Migrate by hand. |
Temporal ... is older than 1.29 | Upgrade to BullSequana AI 1.4.0 first. |
the Temporal database is unreachable or the Temporal visibility database is unreachable | Make the CNPG pooler service and the Secret temporal-postgresql-secret work for the temporal role, then sync again. |
Temporal ... expects core schema 1.18 before 1.30.7, but the database has ... | The server version and the database disagree. Find out which server last wrote to the database before you change anything. If the schema is newer, a manual migration ran partway: continue it by hand. |
Temporal ... is deployed but its database has no core schema | The Deployments point at a database without Temporal tables. Check the database names in variables.toml against the database the 1.4.0 server used. |
the Temporal server Deployments are missing but the core schema is ... | The Deployments were deleted. Migrate by hand. |
Migrate by hand
Use this path for an installation the Job does not recognise, or when you set TEMPORAL_MIGRATION_MODE=off to control each release yourself. Do it before you sync 2.0.0. With the mode off, the chart's own schema Job would move a 1.29 schema straight to 1.32.
-
Pause automated sync on the
temporalApplication, and take the backup above. -
Render the platform with
TEMPORAL_MIGRATION_MODE=auto. The filecoreai/temporal/extra/temporal-migration.yamlcontains the ConfigMaptemporal-migration, with the migration script and the configuration each intermediate server uses. Apply the ServiceAccount, Role, RoleBinding and ConfigMap from that file. Do not apply its Jobs. -
Stop the server, and wait until its pods are gone:
for svc in frontend history matching worker; do kubectl -n temporal scale deploy temporal-$svc --replicas=0; done kubectl -n temporal get pods -l 'app.kubernetes.io/component in (frontend,history,matching,worker)' -
Wait 30 seconds after the last pod is gone. A starting server tries to reach every server that reported in during the last 20 seconds. The address of a stopped pod can delay that past the server's 30-second limit, and the server then exits.
-
For each release, first
1.30.7and then1.31.3, run the steps the Job runs. Create a Pod from thetemporal-migrationJob in the rendered file, and keep only its containersfetch-server-<release>andstep-<release>. Add an init container that writes the release to/work/plan:sh -c 'echo 1.30.7 > /work/plan'Wait for the Pod to finish, read the log of
step-<release>, and delete the Pod before you start the next release. The step updates the core and visibility schemas with that release'sadmin-toolsimage. It then runs that release's server until it serves requests on the frontend, history and matching services, and stops it after 60 seconds. -
Record the result, so a later automatic run does nothing:
kubectl -n temporal create configmap temporal-migration-status \ --from-literal=phase=done --from-literal=from=1.29.6.1 \ --from-literal=steps=1.30.7,1.31.3 --from-literal=to=1.31.3 -
Sync 2.0.0. The chart deploys 1.32.0, and its schema Job finds the schema already current.
-
Delete the Elasticsearch volumes after Argo CD has removed the StatefulSet
elasticsearch-master:kubectl -n temporal delete pvc -l app=elasticsearch-master
Restore Temporal 1.4.0 from the backup
Restore only when you must return to Temporal 1.29. Workflows that changed after the backup lose those changes.
-
Pause automated sync on the
temporalApplication, and point it back at the 1.4.0 manifests. -
Scale the four server Deployments to zero, and wait until their pods are gone.
-
Replace the contents of both databases with the backup files. CNPG owns the databases themselves, so keep them and recreate their
publicschema. Run this against the CNPG primary:PRIMARY=$(kubectl -n cnpg get cluster cnpg-cluster -o jsonpath='{.status.currentPrimary}') for db in temporal temporal_visibility; do kubectl -n cnpg exec -i "$PRIMARY" -c postgres -- psql -U postgres -v ON_ERROR_STOP=1 -d "$db" \ -c "DROP SCHEMA public CASCADE" -c "CREATE SCHEMA public AUTHORIZATION temporal" kubectl -n cnpg exec -i "$PRIMARY" -c postgres -- psql -U postgres -v ON_ERROR_STOP=1 -d "$db" \ < "temporal-backup-$db.sql" done -
Delete the status ConfigMap:
kubectl -n temporal delete configmap temporal-migration-status -
Sync the 1.4.0 manifests. They recreate Elasticsearch with an empty index, so search shows only workflows that change after the restore.
Upgrade 1.4.0 to 2.0.0
Upgrade BullSequana AI 1.4.0 to 2.0.0 with the managed upgrade or the manual GitOps path, check the result, and recover from the pre-upgrade backups.
Move models from LiteLLM to model routing
Record LiteLLM's models before the BullSequana AI 2.0.0 upgrade, and serve them again through model routes after it.