External Vault secrets

BullSequana AI can deliver its Kubernetes secrets from a customer-managed HashiCorp Vault-compatible KV store instead of committing sealed secrets to the manifests repository. This playbook is for the platform engineer who deploys a fresh cluster and the Vault owner who provides the secret entries.

With this backend, the Platform CLI replaces every rendered Secret with an ExternalSecret resource. The resource keeps the secret's name, keys, type, and labels but carries no values. External Secrets Operator (ESO) reads the values from the Vault and creates the Secret in the cluster.

The Vault is a prerequisite outside the platform. The Vault owner runs it, backs it up, and controls access to it. BullSequana AI does not deploy, initialize, or populate it.

Choose a secrets backend

With external-vault:

• the sealed-secrets component is not rendered
• apply-sealing-key refuses to run, because there is no sealing key to install
• the external-secrets Argo CD Application moves from sync wave 12 to 1 in the common parent, so ESO runs before the components whose secrets it delivers
• each generated ExternalSecret has sync wave -5 inside its own Application, so the Secret exists before the workloads that consume it
• kubernetes.io/service-account-token secrets stay as plain Secret resources, because Kubernetes fills their values

This playbook covers a fresh deployment. It does not describe how to move an existing cluster from sealed secrets to an external Vault.

1. Prepare the Vault

The Vault owner provides:

• an endpoint that ESO pods can reach from the cluster
• a KV secrets engine mount, kv by default, at version v2 by default
• a token that can read every entry under the platform path
• the CA trust for the endpoint, when HTTPS uses a private certificate chain

Give the token read-only access to the platform entries. ESO only reads secrets, so the token does not need write, list, or delete permissions.

2. Configure the backend

Set the backend in platform.env:

SECRETS_BACKEND=external-vault
SECRETS_VAULT_URL=https://vault.example.com:8200

The other settings have defaults. Override them only when the site needs a different value:

The CLI never renders the token Secret. It only references it from the ClusterSecretStore.

ClusterSecretStore is the only supported store kind.

3. Render and list the required entries

Render the complete desired state, then print the Vault entries that the rendered ExternalSecret resources read:

bsqai apply all -r
bsqai list-vault-secrets

The command prints one line per remote key:

<key>: <property>, <property>, ...

The remote key of a secret is <path_prefix>/<namespace>/<secret-name>. Each property is one data key of the Kubernetes Secret. For example, with an empty prefix, the credential-external-keycloak secret in the keycloak namespace reads properties ADMIN_PASSWORD and ADMIN_USERNAME from entry keycloak/credential-external-keycloak.

A line that ends with * means ESO copies the whole entry, so every property in the Vault entry ends up in the Secret. Supply the value the application should receive, not a second base64 encoding of Kubernetes data. The list contains names, not values.

list-vault-secrets reads the rendered output. It fails when no ExternalSecret is found, for example when the render used the sealed-secrets backend.

4. Populate the Vault

Create every listed entry and property in the Vault before Argo CD syncs the platform. An ExternalSecret whose entry is missing stays unresolved, and the workloads that consume its Secret do not start.

The values come from the site's secret-management process. When RANDOMIZE_SECRETS generates values, generated_secrets.env holds them after the render. Copy them into the Vault and protect that file until the transfer is complete. See Secret values.

Review the generated manifests before pushing them. Confirm that:

• no rendered file contains a Secret with values, except service-account tokens
• every ExternalSecret references the ClusterSecretStore named in SECRETS_STORE_NAME
• the ClusterSecretStore in the external-secrets output points at the intended endpoint, mount, and version
• the sealed-secrets Application is absent from the generated common output

Then publish the desired state:

bsqai push-manifests "Deploy BullSequana AI with an external Vault"

5. Bootstrap the cluster

ESO needs two resources before Argo CD starts it: a registry pull secret for its images and the Vault token. Create both in the external-secrets namespace:

bsqai apply-registry-secret external-secrets
kubectl -n external-secrets create secret generic vault-token --from-file=token=./vault-token

apply-registry-secret creates the namespace when it does not exist. It builds the pull secret from the primary registry settings and the optional COMMON_DOCKER_CONFIG_JSON.

Read the token from a protected file rather than typing it on the command line, then delete the file. The Secret name, namespace, and key must match the SECRETS_VAULT_TOKEN_SECRET_* settings.

Bootstrap Argo CD:

bsqai helm-apply argocd

Do not run apply-sealing-key with this backend.

6. Verify after deployment

Check that the store is ready and that every ExternalSecret has synced:

kubectl get clustersecretstore
kubectl get externalsecret -A

The store reports Valid and each ExternalSecret reports SecretSynced. Then continue with the acceptance checks in the deployment sequence.

Existing deployment cutover

Preserve the deployed credential values and their ownership when changing backends. Coordinate retirement of the old SealedSecret owner with adoption by External Secrets Operator so two controllers do not compete for the same Secret. Keep secret backups and sealing keys until the cutover and recovery path have been verified.

After reconciliation, check that target Secret keys exist and a dependent service authenticates successfully. Inspect status and key names without printing credential values.

Rotate a secret

Update the value in the Vault. ESO writes the new value into the Kubernetes Secret at the next refresh, within SECRETS_REFRESH_INTERVAL. Workloads that read the secret only at start-up need a restart to use the new value.

When the Vault token changes, update the token Secret in the external-secrets namespace.

Troubleshooting

• ClusterSecretStore is not Valid: check the endpoint, the CA trust, the KV mount and version, and that the token Secret exists with the configured name, namespace, and key.
• An ExternalSecret reports a missing key or property: compare the Vault entry with the output of bsqai list-vault-secrets.
• ESO pods are in ImagePullBackOff: run bsqai apply-registry-secret external-secrets and check the registry credentials.
• Access is denied: check that the token policy allows read access to every path under the platform prefix.
• apply-sealing-key exits with an error: this is expected with external-vault. Skip that step.

Related pages

• Configuration model
• Platform CLI
• Deployment sequence
• External Secrets Operator