PostgreSQL Backup and Disaster Recovery

This playbook explains how the platform's shared CNPG PostgreSQL cluster is backed up, how backup credentials are rotated safely, and what to check when planning a restore.

Backup Model

The CNPG cluster backs up to an S3-compatible bucket on Rook-Ceph RGW using CNPG's barmanObjectStore method. Backups are controlled through two pieces of configuration in the cnpg component's [backups] table:

[backups]
enabled           = true
schedule          = "0 0 2 * * *"  # daily at 02:00
name              = "daily-backup"
retention         = "30d"
access_key_id     = {value = "", secret = true, generate = "s3_access_key_id"}
access_secret_key = {value = "", secret = true, generate = "s3_secret_access_key"}

A ScheduledBackup custom resource is rendered from this configuration and applied at ArgoCD sync-wave 2, ahead of the cluster's other post-sync hooks. It intentionally omits spec.immediate — the CNPG Helm chart hardcodes immediate: true, which would trigger an unwanted backup on every sync; managing the ScheduledBackup as a separate manifest avoids that.

Backup credentials (access_key_id / access_secret_key) are generated automatically if left empty and are propagated to Rook-Ceph as a dedicated cnpgbackups object-store user via Reflector.

Credential Rotation Safety

Because the backup S3 credentials are shared between the CNPG Cluster resource and the underlying Rook-Ceph object-store user, changing them requires the two to be updated in lockstep. A PostSync hook Job (cnpg-backup-rotation) automates this:

1. On every sync, it hashes the current ACCESS_KEY_ID / ACCESS_SECRET_KEY in the cnpg-backups-s3-credentials secret.
2. On first install, it records that hash on the Cluster resource and exits without restarting anything.
3. On later syncs, if the hash is unchanged, it is a no-op.
4. If the hash has changed (the credentials were rotated), it waits for Reflector to mirror the new secret into the rook-ceph namespace, waits for the corresponding CephObjectStoreUser to reach Ready, and only then triggers a CNPG-managed rolling restart (via the kubectl.kubernetes.io/restartedAt annotation) so the cluster picks up the new credentials without a manual, unsynchronized restart.
5. It waits for the cluster to report Cluster in healthy state with all instances ready before recording the new hash and completing.

This ordering exists specifically to avoid a rolling restart being triggered before Rook-Ceph has actually finished propagating rotated credentials, which would otherwise leave CNPG unable to reach its own backup target mid-restart.

Retention

Backups are retained according to the retention value (30d by default), enforced by CNPG's own retention policy against the barmanObjectStore. Retention is a property of the Cluster resource, so changing it takes effect on the next reconcile without requiring a new backup to be taken immediately.

Restore Considerations

CNPG restores are performed by bootstrapping a new Cluster from an existing backup (CNPG does not support in-place restore of a running cluster). Before performing a restore:

1. Confirm which components share this CNPG instance — the shared pooler is used by Keycloak, Grafana, LiteLLM, the CoreAI LLM backend, MLflow, Airbyte, Superset so a restore affects all of their state simultaneously; there is no per-database restore path at the CNPG level.
2. Confirm the target recovery point is covered by the retention window.
3. Confirm the cnpg-backups-s3-credentials secret used for the restore matches the credentials active on the Rook-Ceph object-store user at the time of the backup being restored, not necessarily the current ones if a rotation happened since.

Related Pages

• CNPG-backed PostgreSQL
• Rook-Ceph
• Reflector
• Configuration model