GitOps Workflow

How the platform uses ArgoCD and the app-of-apps pattern to deploy from Git.

Agentic Friendly

BullSequana AI uses a GitOps deployment model. The bsqai-platform CLI generates Kubernetes manifests locally and pushes them to a Git repository. ArgoCD, running inside the cluster, watches that repository and reconciles the desired state automatically.

The only component installed directly on the cluster is ArgoCD itself. Everything else is deployed through Git.

Manifest generation flow

  1. The CLI loads global config from platform.toml and component config from each variables.toml
  2. Environment variables override any resolved value
  3. Jinja2 renders values.yaml.j2 and any resource/secret templates
  4. Any rendered kind: Secret manifest is automatically sealed with kubeseal
  5. An ArgoCD Application manifest is generated for each component
  6. Parent app-of-apps manifests are generated for each group
  7. The full output is pushed to the manifests Git repository
  8. ArgoCD detects the change and deploys

App-of-apps pattern

The platform uses ArgoCD's app-of-apps pattern to organize deployment into three tiers. Each tier is a parent ArgoCD Application that manages all child component Application manifests within its directory.

Parent applications

Parent appSync waveComponents
common1Cluster infrastructure: cert-manager, Keycloak, Grafana stack, sealed-secrets, PostgreSQL, storage, networking
coreai2AI platform: Portal, LLM Backend, LiteLLM, KubeAI, Milvus, MLflow, Temporal, model serving
proai3Data and ML extensions: Superset, Airbyte, Argo Workflows, Argo Events

Sync waves ensure infrastructure deploys before application-layer components. Within each parent app, individual components also have their own sync wave (0–99) to control ordering.

Child application structure

Each component gets an ArgoCD Application manifest that defines:

  • Sources: the Helm chart from the OCI registry, the rendered values.yaml from Git, and optionally extra templates from Git
  • Destination: the component's namespace on the cluster
  • Sync policy: automated pruning, self-healing, server-side apply, retry with exponential backoff
  • Ignore differences: component-specific rules to prevent sync drift from controller-mutated fields

For Helm components, the ArgoCD Application uses multi-source: one source points to the chart in the OCI registry, another points to the rendered values file in the Git manifests repo.

Output directory structure

After running apply all -r (the -r flag replaces the existing output directory before generating), the output directory looks like this:

src/manifests/output/
├── common.yaml              # Parent app-of-apps for common
├── coreai.yaml              # Parent app-of-apps for coreai
├── proai.yaml               # Parent app-of-apps for proai
├── common/
│   ├── keycloak-argoapp.yaml
│   ├── keycloak/
│   │   ├── values.yaml
│   │   └── extra/
│   │       ├── keycloak-sso-config-secret.yaml   # sealed
│   │       └── keycloak-client-setup.yaml
│   ├── grafana-argoapp.yaml
│   ├── grafana/
│   │   └── values.yaml
│   └── ...
├── coreai/
│   ├── portal-argoapp.yaml
│   ├── portal/
│   │   └── values.yaml
│   └── ...
└── proai/
    └── ...

The parent YAML files (common.yaml, coreai.yaml, proai.yaml) are the app-of-apps entries. Each component directory contains its rendered values.yaml and optionally an extra/ directory with additional resources and sealed secrets.

Secret sealing

Secrets are never stored in Git as plaintext. The CLI seals them automatically during manifest generation.

How it works

  1. If no .crt file is provided in src/manifests/seal/, the CLI generates a full keypair (certificate + private key)
  2. The apply-sealing-key command pushes the keypair to the cluster, where only the sealed-secrets controller can access the private key
  3. During apply, any rendered manifest containing kind: Secret is encrypted using the certificate via kubeseal
  4. The output is a SealedSecret resource that ArgoCD can safely apply
  5. The sealed-secrets controller in the cluster decrypts it back into a regular Secret

Bootstrap

Before the first deployment, the sealing keypair must be installed on the cluster:

uv run -m src.cli.main apply-sealing-key sealed-secrets

The argument is the target namespace (defaults to sealed-secrets if omitted). This installs the sealed-secrets TLS keypair and the container registry pull secret.

Sync policy

All ArgoCD Applications use the same sync policy:

SettingValue
Automated syncEnabled
PruningEnabled
Self-healEnabled
Server-side applyEnabled
Create namespaceEnabled
Retry limit50 attempts
Retry backoff5s initial, factor 2, max 1m

This means ArgoCD automatically detects drift from the Git state and corrects it. Manual intervention is only needed when a sync fails repeatedly.

SSO enrollment lifecycle

Components with SSO enabled follow a specific lifecycle during ArgoCD sync:

  1. Negative sync wave: a setup job registers the OIDC client with Keycloak before the component itself deploys, using a lower sync wave number to guarantee ordering
  2. Component sync wave: the component is deployed with SSO configuration already in place
  3. PostDelete: a cleanup job removes the Keycloak client when the component's ArgoCD Application is deleted

This is handled by three shared templates in src/manifests/source/template/:

  • keycloak-sso-config-secret.yaml.j2 — the sealed OIDC client configuration
  • keycloak-client-setup.yaml.j2 — the registration job (runs at a negative sync wave, before the component)
  • keycloak-client-cleanup.yaml.j2 — the PostDelete cleanup job

On this page