GitOps Workflow
How the platform uses ArgoCD and the app-of-apps pattern to deploy from Git.
BullSequana AI uses a GitOps deployment model. The bsqai-platform CLI generates Kubernetes manifests locally and pushes them to a Git repository. ArgoCD, running inside the cluster, watches that repository and reconciles the desired state automatically.
The only component installed directly on the cluster is ArgoCD itself. Everything else is deployed through Git.
Manifest generation flow
- The CLI loads global config from
platform.tomland component config from eachvariables.toml - Environment variables override any resolved value
- Jinja2 renders
values.yaml.j2and any resource/secret templates - Any rendered
kind: Secretmanifest is automatically sealed withkubeseal - An ArgoCD
Applicationmanifest is generated for each component - Parent app-of-apps manifests are generated for each group
- The full output is pushed to the manifests Git repository
- ArgoCD detects the change and deploys
App-of-apps pattern
The platform uses ArgoCD's app-of-apps pattern to organize deployment into three tiers. Each tier is a parent ArgoCD Application that manages all child component Application manifests within its directory.
Parent applications
| Parent app | Sync wave | Components |
|---|---|---|
common | 1 | Cluster infrastructure: cert-manager, Keycloak, Grafana stack, sealed-secrets, PostgreSQL, storage, networking |
coreai | 2 | AI platform: Portal, LLM Backend, LiteLLM, KubeAI, Milvus, MLflow, Temporal, model serving |
proai | 3 | Data and ML extensions: Superset, Airbyte, Argo Workflows, Argo Events |
Sync waves ensure infrastructure deploys before application-layer components. Within each parent app, individual components also have their own sync wave (0–99) to control ordering.
Child application structure
Each component gets an ArgoCD Application manifest that defines:
- Sources: the Helm chart from the OCI registry, the rendered
values.yamlfrom Git, and optionally extra templates from Git - Destination: the component's namespace on the cluster
- Sync policy: automated pruning, self-healing, server-side apply, retry with exponential backoff
- Ignore differences: component-specific rules to prevent sync drift from controller-mutated fields
For Helm components, the ArgoCD Application uses multi-source: one source points to the chart in the OCI registry, another points to the rendered values file in the Git manifests repo.
Output directory structure
After running apply all -r (the -r flag replaces the existing output directory before generating), the output directory looks like this:
src/manifests/output/
├── common.yaml # Parent app-of-apps for common
├── coreai.yaml # Parent app-of-apps for coreai
├── proai.yaml # Parent app-of-apps for proai
├── common/
│ ├── keycloak-argoapp.yaml
│ ├── keycloak/
│ │ ├── values.yaml
│ │ └── extra/
│ │ ├── keycloak-sso-config-secret.yaml # sealed
│ │ └── keycloak-client-setup.yaml
│ ├── grafana-argoapp.yaml
│ ├── grafana/
│ │ └── values.yaml
│ └── ...
├── coreai/
│ ├── portal-argoapp.yaml
│ ├── portal/
│ │ └── values.yaml
│ └── ...
└── proai/
└── ...The parent YAML files (common.yaml, coreai.yaml, proai.yaml) are the app-of-apps entries. Each component directory contains its rendered values.yaml and optionally an extra/ directory with additional resources and sealed secrets.
Secret sealing
Secrets are never stored in Git as plaintext. The CLI seals them automatically during manifest generation.
How it works
- If no
.crtfile is provided insrc/manifests/seal/, the CLI generates a full keypair (certificate + private key) - The
apply-sealing-keycommand pushes the keypair to the cluster, where only the sealed-secrets controller can access the private key - During
apply, any rendered manifest containingkind: Secretis encrypted using the certificate viakubeseal - The output is a
SealedSecretresource that ArgoCD can safely apply - The sealed-secrets controller in the cluster decrypts it back into a regular
Secret
Bootstrap
Before the first deployment, the sealing keypair must be installed on the cluster:
uv run -m src.cli.main apply-sealing-key sealed-secretsThe argument is the target namespace (defaults to sealed-secrets if omitted). This installs the sealed-secrets TLS keypair and the container registry pull secret.
Sync policy
All ArgoCD Applications use the same sync policy:
| Setting | Value |
|---|---|
| Automated sync | Enabled |
| Pruning | Enabled |
| Self-heal | Enabled |
| Server-side apply | Enabled |
| Create namespace | Enabled |
| Retry limit | 50 attempts |
| Retry backoff | 5s initial, factor 2, max 1m |
This means ArgoCD automatically detects drift from the Git state and corrects it. Manual intervention is only needed when a sync fails repeatedly.
SSO enrollment lifecycle
Components with SSO enabled follow a specific lifecycle during ArgoCD sync:
- Negative sync wave: a setup job registers the OIDC client with Keycloak before the component itself deploys, using a lower sync wave number to guarantee ordering
- Component sync wave: the component is deployed with SSO configuration already in place
- PostDelete: a cleanup job removes the Keycloak client when the component's ArgoCD Application is deleted
This is handled by three shared templates in src/manifests/source/template/:
keycloak-sso-config-secret.yaml.j2— the sealed OIDC client configurationkeycloak-client-setup.yaml.j2— the registration job (runs at a negative sync wave, before the component)keycloak-client-cleanup.yaml.j2— the PostDelete cleanup job