# Upgrade Temporal to 1.32 (/docs/deployment/playbooks/upgrade-temporal-1-32)



BullSequana AI 1.4.0 runs Temporal server 1.29 with Elasticsearch for workflow search. BullSequana AI 2.0.0 runs Temporal 1.32 with workflow search in PostgreSQL. Temporal upgrades one minor release at a time, so 2.0.0 runs 1.30 and 1.31 against your database before it deploys 1.32.

By default the `temporal` Argo CD Application does this itself. Its PreSync Job `temporal-migration` checks the running Temporal and then does one of three things:

* On a fresh install, a cluster already on 1.31 or later, or a re-sync, it does nothing.
* On a standard 1.4.0 Temporal, it stops the 1.29 server and runs 1.30.7, then 1.31.3. Each release updates both schemas and then runs as one process until it is healthy. The chart then deploys 1.32.0.
* On any other state, it changes nothing and fails the sync. Temporal 1.4.0 keeps serving.

Temporal is unavailable for a few minutes during the migration. Workflow state is kept. Workflow search history from 1.4.0 is not carried over: Elasticsearch kept it for three days, and the 2.0.0 sync removes Elasticsearch and its volumes.

The commands below use the default names: namespace `temporal`, release `temporal`, and databases `temporal` and `temporal_visibility` in the CNPG cluster `cnpg-cluster`. Replace them if your installation overrides them.

The [managed upgrade](/docs/deployment/playbooks/upgrade-and-release-process#managed-upgrade) backs up the Temporal databases before the `temporal` sync and checks that the migration completed. On the [manual GitOps upgrade](/docs/deployment/playbooks/upgrade-and-release-process#manual-gitops-upgrade), take the backup below yourself. Do not push the 2.0.0 manifests onto the branch the cluster syncs from while automated sync is on: the `temporal` application could then sync the new values with the old chart before its parent application updates the chart version. The manual upgrade holds reconciliation with a deny sync window first.

## Back up the Temporal databases first [#back-up-the-temporal-databases-first]

On the manual path, take this backup before you sync 2.0.0.

```bash
PRIMARY=$(kubectl -n cnpg get cluster cnpg-cluster -o jsonpath='{.status.currentPrimary}')
for db in temporal temporal_visibility; do
  kubectl -n cnpg exec "$PRIMARY" -c postgres -- pg_dump -U postgres -d "$db" > "temporal-backup-$db.sql"
  grep -q "PostgreSQL database dump complete" "temporal-backup-$db.sql" || echo "INCOMPLETE: $db"
done
```

Keep both files until 2.0.0 runs correctly. In 1.4.0 the `temporal_visibility` database is empty, because search ran in Elasticsearch.

The 2.0.0 sync deletes the Elasticsearch volume that held workflow search. If you want to keep that search history, back up the PersistentVolumeClaim `elasticsearch-master-elasticsearch-master-0` in the `temporal` namespace with your storage tooling before you sync. Temporal 2.0.0 does not read it back.

## Check the migration [#check-the-migration]

The Job records its progress in the ConfigMap `temporal-migration-status`:

```bash
kubectl -n temporal get configmap temporal-migration-status -o jsonpath='{.data}'
```

| Field                            | Meaning                                                                 |
| -------------------------------- | ----------------------------------------------------------------------- |
| `phase`                          | `in-progress` while releases are running, `done` after the last one     |
| `from`                           | the server image tag found before the migration, for example `1.29.6.1` |
| `steps`                          | the releases that finished, in order, for example `1.30.7,1.31.3`       |
| `to`                             | the last release the migration ran                                      |
| `started`, `updated`, `finished` | UTC timestamps                                                          |

Read what the Job decided and did:

```bash
kubectl -n temporal logs job/temporal-migration -c plan
kubectl -n temporal logs job/temporal-migration -c step-1-30-7
kubectl -n temporal logs job/temporal-migration -c step-1-31-3
```

After the sync, the four server Deployments run `1.32.0`, and the PostSync Job `temporal-elasticsearch-cleanup` has deleted the Elasticsearch volumes:

```bash
kubectl -n temporal get deploy -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image
kubectl -n temporal get pvc -l app=elasticsearch-master
```

## Resume a partial migration [#resume-a-partial-migration]

A migration that stops partway leaves `phase: in-progress`. The old server Deployments stay at zero replicas, so Temporal is down until the migration finishes.

1. Find the release that failed. The failed container is the last `step-*` container with a non-zero exit code.

   ```bash
   kubectl -n temporal get pods -l job-name=temporal-migration \
     -o jsonpath='{range .items[*].status.initContainerStatuses[*]}{.name}={.state.terminated.exitCode}{"\n"}{end}'
   ```

2. Read that container's log. On a failure it prints the last lines of the server log.

3. Fix the cause. A database that does not accept connections is the usual cause. Check the CNPG cluster and the Secret `temporal-postgresql-secret`.

4. Sync the `temporal` Application again. The new Job reads `steps`, skips the releases that finished, and runs the rest. A release that stopped halfway runs again from its schema update. The schema tools apply only the missing changes.

Never set the Deployments back to 1.29 after a release has run. Temporal 1.29 cannot run on a schema that 1.30 or 1.31 updated. To go back to 1.4.0, restore the backup.

## After the migration stopped without changing anything [#after-the-migration-stopped-without-changing-anything]

The sync fails with `Temporal migration stopped before changing anything:` and a reason. Temporal 1.4.0 keeps serving, and the ConfigMap is not created. Argo CD retries the sync, and the Job stops again until the state matches what it recognises.

| Reason in the message                                                                       | What to do                                                                                                                                                                                              |
| ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `the Temporal server Deployments are incomplete or run different image tags`                | Finish or roll back the change on the Deployments, so that `temporal-frontend`, `-history`, `-matching` and `-worker` all exist and run one tag. Then sync again.                                       |
| `the Temporal server image tag ... is not a release version`                                | The Deployments run a custom tag or digest. Migrate by hand.                                                                                                                                            |
| `Temporal ... is older than 1.29`                                                           | Upgrade to BullSequana AI 1.4.0 first.                                                                                                                                                                  |
| `the Temporal database is unreachable` or `the Temporal visibility database is unreachable` | Make the CNPG pooler service and the Secret `temporal-postgresql-secret` work for the `temporal` role, then sync again.                                                                                 |
| `Temporal ... expects core schema 1.18 before 1.30.7, but the database has ...`             | The server version and the database disagree. Find out which server last wrote to the database before you change anything. If the schema is newer, a manual migration ran partway: continue it by hand. |
| `Temporal ... is deployed but its database has no core schema`                              | The Deployments point at a database without Temporal tables. Check the database names in `variables.toml` against the database the 1.4.0 server used.                                                   |
| `the Temporal server Deployments are missing but the core schema is ...`                    | The Deployments were deleted. Migrate by hand.                                                                                                                                                          |

## Migrate by hand [#migrate-by-hand]

Use this path for an installation the Job does not recognise, or when you set `TEMPORAL_MIGRATION_MODE=off` to control each release yourself. Do it before you sync 2.0.0. With the mode `off`, the chart's own schema Job would move a 1.29 schema straight to 1.32.

1. Pause automated sync on the `temporal` Application, and take the backup above.

2. Render the platform with `TEMPORAL_MIGRATION_MODE=auto`. The file `coreai/temporal/extra/temporal-migration.yaml` contains the ConfigMap `temporal-migration`, with the migration script and the configuration each intermediate server uses. Apply the ServiceAccount, Role, RoleBinding and ConfigMap from that file. Do not apply its Jobs.

3. Stop the server, and wait until its pods are gone:

   ```bash
   for svc in frontend history matching worker; do kubectl -n temporal scale deploy temporal-$svc --replicas=0; done
   kubectl -n temporal get pods -l 'app.kubernetes.io/component in (frontend,history,matching,worker)'
   ```

4. Wait 30 seconds after the last pod is gone. A starting server tries to reach every server that reported in during the last 20 seconds. The address of a stopped pod can delay that past the server's 30-second limit, and the server then exits.

5. For each release, first `1.30.7` and then `1.31.3`, run the steps the Job runs. Create a Pod from the `temporal-migration` Job in the rendered file, and keep only its containers `fetch-server-<release>` and `step-<release>`. Add an init container that writes the release to `/work/plan`:

   ```bash
   sh -c 'echo 1.30.7 > /work/plan'
   ```

   Wait for the Pod to finish, read the log of `step-<release>`, and delete the Pod before you start the next release. The step updates the core and visibility schemas with that release's `admin-tools` image. It then runs that release's server until it serves requests on the frontend, history and matching services, and stops it after 60 seconds.

6. Record the result, so a later automatic run does nothing:

   ```bash
   kubectl -n temporal create configmap temporal-migration-status \
     --from-literal=phase=done --from-literal=from=1.29.6.1 \
     --from-literal=steps=1.30.7,1.31.3 --from-literal=to=1.31.3
   ```

7. Sync 2.0.0. The chart deploys 1.32.0, and its schema Job finds the schema already current.

8. Delete the Elasticsearch volumes after Argo CD has removed the StatefulSet `elasticsearch-master`:

   ```bash
   kubectl -n temporal delete pvc -l app=elasticsearch-master
   ```

## Restore Temporal 1.4.0 from the backup [#restore-temporal-140-from-the-backup]

Restore only when you must return to Temporal 1.29. Workflows that changed after the backup lose those changes.

1. Pause automated sync on the `temporal` Application, and point it back at the 1.4.0 manifests.

2. Scale the four server Deployments to zero, and wait until their pods are gone.

3. Replace the contents of both databases with the backup files. CNPG owns the databases themselves, so keep them and recreate their `public` schema. Run this against the CNPG primary:

   ```bash
   PRIMARY=$(kubectl -n cnpg get cluster cnpg-cluster -o jsonpath='{.status.currentPrimary}')
   for db in temporal temporal_visibility; do
     kubectl -n cnpg exec -i "$PRIMARY" -c postgres -- psql -U postgres -v ON_ERROR_STOP=1 -d "$db" \
       -c "DROP SCHEMA public CASCADE" -c "CREATE SCHEMA public AUTHORIZATION temporal"
     kubectl -n cnpg exec -i "$PRIMARY" -c postgres -- psql -U postgres -v ON_ERROR_STOP=1 -d "$db" \
       < "temporal-backup-$db.sql"
   done
   ```

4. Delete the status ConfigMap:

   ```bash
   kubectl -n temporal delete configmap temporal-migration-status
   ```

5. Sync the 1.4.0 manifests. They recreate Elasticsearch with an empty index, so search shows only workflows that change after the restore.
