# External Vault secrets (/docs/deployment/playbooks/external-vault-secrets) BullSequana AI can deliver its Kubernetes secrets from a customer-managed HashiCorp Vault-compatible KV store instead of committing sealed secrets to the manifests repository. This playbook is for the platform engineer who deploys a fresh cluster and the Vault owner who provides the secret entries. With this backend, the Platform CLI replaces every rendered `Secret` with an `ExternalSecret` resource. The resource keeps the secret's name, keys, type, and labels but carries no values. External Secrets Operator (ESO) reads the values from the Vault and creates the `Secret` in the cluster. The Vault is a prerequisite outside the platform. The Vault owner runs it, backs it up, and controls access to it. BullSequana AI does not deploy, initialize, or populate it. ## Choose a secrets backend [#choose-a-secrets-backend] | Backend | Where secret values live | What the manifests repository holds | | -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------ | | `sealed-secrets` (default) | Encrypted in Git, decrypted by the sealed-secrets controller | `SealedSecret` resources | | `external-vault` | In the external Vault | `ExternalSecret` resources and one `ClusterSecretStore`, with no secret values | With `external-vault`: * the `sealed-secrets` component is not rendered * `apply-sealing-key` refuses to run, because there is no sealing key to install * the `external-secrets` Argo CD Application moves from sync wave `12` to `1` in the `common` parent, so ESO runs before the components whose secrets it delivers * each generated `ExternalSecret` has sync wave `-5` inside its own Application, so the `Secret` exists before the workloads that consume it * `kubernetes.io/service-account-token` secrets stay as plain `Secret` resources, because Kubernetes fills their values This playbook covers a fresh deployment. It does not describe how to move an existing cluster from sealed secrets to an external Vault. ## 1. Prepare the Vault [#1-prepare-the-vault] The Vault owner provides: * an endpoint that ESO pods can reach from the cluster * a KV secrets engine mount, `kv` by default, at version `v2` by default * a token that can read every entry under the platform path * the CA trust for the endpoint, when HTTPS uses a private certificate chain Give the token read-only access to the platform entries. ESO only reads secrets, so the token does not need write, list, or delete permissions. ## 2. Configure the backend [#2-configure-the-backend] Set the backend in `platform.env`: ```bash SECRETS_BACKEND=external-vault SECRETS_VAULT_URL=https://vault.example.com:8200 ``` The other settings have defaults. Override them only when the site needs a different value: | Variable | Default | Description | | -------------------------------------- | ------------------ | ----------------------------------------------------------------------- | | `SECRETS_BACKEND` | `sealed-secrets` | `sealed-secrets` or `external-vault`. An unknown value fails the render | | `SECRETS_STORE_NAME` | `vault-store` | Name of the `ClusterSecretStore` that every `ExternalSecret` references | | `SECRETS_PATH_PREFIX` | *(empty)* | Prefix of every remote key in the Vault | | `SECRETS_REFRESH_INTERVAL` | `1h` | How often ESO reads each entry again | | `SECRETS_VAULT_URL` | *(empty)* | Vault endpoint. Required with `external-vault` | | `SECRETS_VAULT_PATH` | `kv` | KV secrets engine mount | | `SECRETS_VAULT_VERSION` | `v2` | KV secrets engine version | | `SECRETS_VAULT_TOKEN_SECRET_NAME` | `vault-token` | Kubernetes `Secret` that holds the Vault token | | `SECRETS_VAULT_TOKEN_SECRET_NAMESPACE` | `external-secrets` | Namespace of that `Secret` | | `SECRETS_VAULT_TOKEN_SECRET_KEY` | `token` | Key inside that `Secret` | The CLI never renders the token `Secret`. It only references it from the `ClusterSecretStore`. `ClusterSecretStore` is the only supported store kind. ## 3. Render and list the required entries [#3-render-and-list-the-required-entries] Render the complete desired state, then print the Vault entries that the rendered `ExternalSecret` resources read: ```bash bsqai apply all -r bsqai list-vault-secrets ``` The command prints one line per remote key: ```text : , , ... ``` The remote key of a secret is `//`. Each property is one data key of the Kubernetes `Secret`. For example, with an empty prefix, the `credential-external-keycloak` secret in the `keycloak` namespace reads properties `ADMIN_PASSWORD` and `ADMIN_USERNAME` from entry `keycloak/credential-external-keycloak`. A line that ends with `*` means ESO copies the whole entry, so every property in the Vault entry ends up in the `Secret`. Supply the value the application should receive, not a second base64 encoding of Kubernetes `data`. The list contains names, not values. `list-vault-secrets` reads the rendered output. It fails when no `ExternalSecret` is found, for example when the render used the `sealed-secrets` backend. ## 4. Populate the Vault [#4-populate-the-vault] Create every listed entry and property in the Vault before Argo CD syncs the platform. An `ExternalSecret` whose entry is missing stays unresolved, and the workloads that consume its `Secret` do not start. The values come from the site's secret-management process. When `RANDOMIZE_SECRETS` generates values, `generated_secrets.env` holds them after the render. Copy them into the Vault and protect that file until the transfer is complete. See [Secret values](/docs/deployment/configuration-model#secret-values). Review the generated manifests before pushing them. Confirm that: * no rendered file contains a `Secret` with values, except service-account tokens * every `ExternalSecret` references the `ClusterSecretStore` named in `SECRETS_STORE_NAME` * the `ClusterSecretStore` in the `external-secrets` output points at the intended endpoint, mount, and version * the `sealed-secrets` Application is absent from the generated `common` output Then publish the desired state: ```bash bsqai push-manifests "Deploy BullSequana AI with an external Vault" ``` ## 5. Bootstrap the cluster [#5-bootstrap-the-cluster] ESO needs two resources before Argo CD starts it: a registry pull secret for its images and the Vault token. Create both in the `external-secrets` namespace: ```bash bsqai apply-registry-secret external-secrets kubectl -n external-secrets create secret generic vault-token --from-file=token=./vault-token ``` `apply-registry-secret` creates the namespace when it does not exist. It builds the pull secret from the primary registry settings and the optional `COMMON_DOCKER_CONFIG_JSON`. Read the token from a protected file rather than typing it on the command line, then delete the file. The `Secret` name, namespace, and key must match the `SECRETS_VAULT_TOKEN_SECRET_*` settings. Bootstrap Argo CD: ```bash bsqai helm-apply argocd ``` Do not run `apply-sealing-key` with this backend. ## 6. Verify after deployment [#6-verify-after-deployment] Check that the store is ready and that every `ExternalSecret` has synced: ```bash kubectl get clustersecretstore kubectl get externalsecret -A ``` The store reports `Valid` and each `ExternalSecret` reports `SecretSynced`. Then continue with the acceptance checks in the [deployment sequence](/docs/deployment/playbooks/deployment-sequence#7-reconcile-and-verify). ## Existing deployment cutover [#existing-deployment-cutover] Preserve the deployed credential values and their ownership when changing backends. Coordinate retirement of the old SealedSecret owner with adoption by External Secrets Operator so two controllers do not compete for the same Secret. Keep secret backups and sealing keys until the cutover and recovery path have been verified. After reconciliation, check that target Secret keys exist and a dependent service authenticates successfully. Inspect status and key names without printing credential values. ## Rotate a secret [#rotate-a-secret] Update the value in the Vault. ESO writes the new value into the Kubernetes `Secret` at the next refresh, within `SECRETS_REFRESH_INTERVAL`. Workloads that read the secret only at start-up need a restart to use the new value. When the Vault token changes, update the token `Secret` in the `external-secrets` namespace. ## Troubleshooting [#troubleshooting] * `ClusterSecretStore` is not `Valid`: check the endpoint, the CA trust, the KV mount and version, and that the token `Secret` exists with the configured name, namespace, and key. * An `ExternalSecret` reports a missing key or property: compare the Vault entry with the output of `bsqai list-vault-secrets`. * ESO pods are in `ImagePullBackOff`: run `bsqai apply-registry-secret external-secrets` and check the registry credentials. * Access is denied: check that the token policy allows read access to every path under the platform prefix. * `apply-sealing-key` exits with an error: this is expected with `external-vault`. Skip that step. ## Related pages [#related-pages] * [Configuration model](/docs/deployment/configuration-model) * [Platform CLI](/docs/deployment/platform-cli) * [Deployment sequence](/docs/deployment/playbooks/deployment-sequence) * [External Secrets Operator](/docs/foundation/components/external-secrets)