# GitOps Workflow (/docs/deployment/gitops-workflow)



BullSequana AI uses a GitOps deployment model. The `bsqai-platform` CLI generates Kubernetes manifests locally and pushes them to a Git repository. ArgoCD, running inside the cluster, watches that repository and reconciles the desired state automatically.

The only component installed directly on the cluster is ArgoCD itself. Everything else is deployed through Git.

Manifest generation flow [#manifest-generation-flow]

<Mermaid
  chart="flowchart TD
    A[platform.toml + variables.toml] --> B[CLI: load and resolve]
    B --> C[Jinja2 template rendering]
    C --> D{Contains Secret?}
    D -- yes --> E[kubeseal encryption]
    D -- no --> F[Write to output/]
    E --> F
    F --> G[Generate ArgoCD Application manifests]
    G --> H[Push to manifests Git repo]
    H --> I[ArgoCD reconciles from Git]"
/>

1. The CLI loads global config from `platform.toml` and component config from each `variables.toml`
2. Environment variables override any resolved value
3. Jinja2 renders `values.yaml.j2` and any resource/secret templates
4. Any rendered `kind: Secret` manifest is automatically sealed with `kubeseal`
5. An ArgoCD `Application` manifest is generated for each component
6. Parent app-of-apps manifests are generated for each group
7. The full output is pushed to the manifests Git repository
8. ArgoCD detects the change and deploys

App-of-apps pattern [#app-of-apps-pattern]

The platform uses ArgoCD's app-of-apps pattern to organize deployment into three tiers. Each tier is a parent ArgoCD `Application` that manages all child component `Application` manifests within its directory.

<Mermaid
  chart="flowchart TD
    A[ArgoCD] --> B[common - wave 1]
    A --> C[coreai - wave 2]
    A --> D[proai - wave 3]
    B --> B1[cert-manager]
    B --> B2[keycloak]
    B --> B3[grafana]
    B --> B4[...]
    C --> C1[portal]
    C --> C2[litellm]
    C --> C3[kubeai]
    C --> C4[...]
    D --> D1[superset]
    D --> D2[airbyte]
    D --> D3[...]"
/>

Parent applications [#parent-applications]

| Parent app | Sync wave | Components                                                                                                     |
| ---------- | --------- | -------------------------------------------------------------------------------------------------------------- |
| `common`   | 1         | Cluster infrastructure: cert-manager, Keycloak, Grafana stack, sealed-secrets, PostgreSQL, storage, networking |
| `coreai`   | 2         | AI platform: Portal, LLM Backend, LiteLLM, KubeAI, Milvus, MLflow, Temporal, model serving                     |
| `proai`    | 3         | Data and ML extensions: Superset, Airbyte, Argo Workflows, Argo Events                                         |

Sync waves ensure infrastructure deploys before application-layer components. Within each parent app, individual components also have their own sync wave (0–99) to control ordering.

Child application structure [#child-application-structure]

Each component gets an ArgoCD `Application` manifest that defines:

* **Sources**: the Helm chart from the OCI registry, the rendered `values.yaml` from Git, and optionally extra templates from Git
* **Destination**: the component's namespace on the cluster
* **Sync policy**: automated pruning, self-healing, server-side apply, retry with exponential backoff
* **Ignore differences**: component-specific rules to prevent sync drift from controller-mutated fields

For Helm components, the ArgoCD Application uses multi-source: one source points to the chart in the OCI registry, another points to the rendered values file in the Git manifests repo.

Output directory structure [#output-directory-structure]

After running `apply all -r` (the `-r` flag replaces the existing output directory before generating), the output directory looks like this:

```text
src/manifests/output/
├── common.yaml              # Parent app-of-apps for common
├── coreai.yaml              # Parent app-of-apps for coreai
├── proai.yaml               # Parent app-of-apps for proai
├── common/
│   ├── keycloak-argoapp.yaml
│   ├── keycloak/
│   │   ├── values.yaml
│   │   └── extra/
│   │       ├── keycloak-sso-config-secret.yaml   # sealed
│   │       └── keycloak-client-setup.yaml
│   ├── grafana-argoapp.yaml
│   ├── grafana/
│   │   └── values.yaml
│   └── ...
├── coreai/
│   ├── portal-argoapp.yaml
│   ├── portal/
│   │   └── values.yaml
│   └── ...
└── proai/
    └── ...
```

The parent YAML files (`common.yaml`, `coreai.yaml`, `proai.yaml`) are the app-of-apps entries. Each component directory contains its rendered `values.yaml` and optionally an `extra/` directory with additional resources and sealed secrets.

Secret sealing [#secret-sealing]

Secrets are never stored in Git as plaintext. The CLI seals them automatically during manifest generation.

How it works [#how-it-works]

1. If no `.crt` file is provided in `src/manifests/seal/`, the CLI generates a full keypair (certificate + private key)
2. The `apply-sealing-key` command pushes the keypair to the cluster, where only the sealed-secrets controller can access the private key
3. During `apply`, any rendered manifest containing `kind: Secret` is encrypted using the certificate via `kubeseal`
4. The output is a `SealedSecret` resource that ArgoCD can safely apply
5. The sealed-secrets controller in the cluster decrypts it back into a regular `Secret`

Bootstrap [#bootstrap]

Before the first deployment, the sealing keypair must be installed on the cluster:

```bash
uv run -m src.cli.main apply-sealing-key sealed-secrets
```

The argument is the target namespace (defaults to `sealed-secrets` if omitted). This installs the sealed-secrets TLS keypair and the container registry pull secret.

Sync policy [#sync-policy]

All ArgoCD Applications use the same sync policy:

| Setting           | Value                        |
| ----------------- | ---------------------------- |
| Automated sync    | Enabled                      |
| Pruning           | Enabled                      |
| Self-heal         | Enabled                      |
| Server-side apply | Enabled                      |
| Create namespace  | Enabled                      |
| Retry limit       | 50 attempts                  |
| Retry backoff     | 5s initial, factor 2, max 1m |

This means ArgoCD automatically detects drift from the Git state and corrects it. Manual intervention is only needed when a sync fails repeatedly.

SSO enrollment lifecycle [#sso-enrollment-lifecycle]

Components with SSO enabled follow a specific lifecycle during ArgoCD sync:

1. **Negative sync wave**: a setup job registers the OIDC client with Keycloak before the component itself deploys, using a lower sync wave number to guarantee ordering
2. **Component sync wave**: the component is deployed with SSO configuration already in place
3. **PostDelete**: a cleanup job removes the Keycloak client when the component's ArgoCD Application is deleted

This is handled by three shared templates in `src/manifests/source/template/`:

* `keycloak-sso-config-secret.yaml.j2` — the sealed OIDC client configuration
* `keycloak-client-setup.yaml.j2` — the registration job (runs at a negative sync wave, before the component)
* `keycloak-client-cleanup.yaml.j2` — the PostDelete cleanup job

Related pages [#related-pages]

* [Platform CLI](/docs/deployment/platform-cli)
* [Configuration model](/docs/deployment/configuration-model)
* [Component anatomy](/docs/deployment/component-anatomy)
* [Deployment sequence](/docs/deployment/playbooks/deployment-sequence)