Troubleshooting

Use this page as the main entry point when something is not working as expected on BullSequana AI.

The goal is simple:

1. identify the failure domain quickly
2. run a few high-signal checks
3. jump to the right section of the documentation

Start With Scope

Before looking at a specific component, confirm what is actually failing:

• a single application
• one platform service
• one namespace
• one environment
• the full platform rollout

That distinction avoids spending time in the wrong layer.

Route By Symptom

Access, login, or permission issues

Check these areas first:

• Security Model
• Keycloak
• OpenFGA
• Use local models via API

Typical questions:

• is the user authenticated successfully
• is the token or API key valid
• is the right endpoint being used
• is authorization blocking access after authentication

Deployment, rollout, or upgrade failures

Check these pages first:

• Deployment
• Configuration Model
• Deployment Sequence
• Upgrade and Release Process
• Harbor Delivery

Typical questions:

• is the cluster configured with the right storage classes
• are registry credentials valid
• is Git pointing to the correct manifests branch and path
• is DNS and certificate configuration complete

A 1.2.1-to-1.3.0 upgrade stopped

Read the durable checkpoint first:

bsqai upgrade status

If the state is failed, correct the reported cause and rerun the same bsqai upgrade apply --from 1.2.1 --to 1.3.0 command. The upgrader reuses its immutable manifest ref and completed Application checkpoints.

Do not delete argocd/bsqai-platform-upgrade, remove the temporary AppProject deny window, restore automated policies, or choose a different manifest ref to force progress. Use --supersede-failed-snapshot only when corrected target manifests require a new immutable snapshot. See Resume a stopped upgrade.

Data Hub connection or ingestion failures

Start with the run's failure code and stage diagnostics in Data Hub. Check the active tenant and resource grants for missing actions, then credentials, image pulls, endpoint access and native source/destination permissions for failed execution. A queued run also depends on the BSQAI API connector worker and Temporal. Use managed-ingestion operations for runtime and checkpoint checks.

Foundation service failures

Check these pages first:

• Foundation
• Observability Model
• Ingress and Routing Model

Typical questions:

• is the failing service healthy
• are its dependencies healthy
• is ingress reaching the service
• is storage, database, or secret access available

Developer integration or application issues

Check these pages first:

• Development
• Use local models via API
• Deploy apps on the platform
• AI, Doc AI & Pipelines

Typical questions:

• is the application using the stable BSQAI API
• is the model name valid in the current environment
• is the bearer token valid
• is the application deployed through the expected GitOps path

First Operational Checks

These checks are useful in almost every incident:

kubectl get ns
kubectl get pods -A
kubectl get httproute -A
kubectl get events -A --sort-by=.lastTimestamp

If the problem is isolated to one namespace:

kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>

Argo CD Tips

Argo CD is often the fastest way to understand whether the issue is in desired state, sync, or runtime behavior.

What to look at

For each affected application, check:

• Health
• Sync Status
• target namespace
• source path and revision
• recent sync operation state

Quick checks:

kubectl get applications -n argocd
kubectl get application <app-name> -n argocd -o yaml

Common Argo CD patterns

OutOfSync

• the live cluster no longer matches Git
• the wrong branch, path, or values may be referenced
• a dependency may have changed without the application being updated

Progressing

• resources are still reconciling
• a dependency is not yet ready
• a hook or sync wave may still be running

Degraded

• the application synced, but some resources are unhealthy
• this usually means the problem has moved from GitOps into workload runtime behavior

Practical Argo CD questions

Ask these in order:

1. Is the application present in Argo CD?
2. Is it Synced?
3. Is it Healthy?
4. If not, which resource is failing?
5. Is the failing resource blocked by secret, database, ingress, or dependency readiness?

Sync-order issues

The platform uses hooks and sync waves in several places. If an application is present but not becoming healthy, check whether:

• prerequisite secrets exist
• the database or storage resource is ready
• the required CRDs are installed
• the application depends on another service that has not finished reconciling

Observability Checks

If the deployment exists but behavior is unclear, move to the observability stack:

• Grafana
• Prometheus
• Grafana Loki
• Grafana Tempo
• Observability Model

Use:

• metrics to confirm health and saturation
• logs to identify failing components
• traces to understand cross-service behavior

Authentication And API Checks

For AI and developer-facing integrations, validate these points early:

• the application is calling the BSQAI API, not internal LiteLLM endpoints
• the model exists in /v1/models
• the token or sk-bsq-... API key is valid
• the failing request is authorized for the current user or key
• a 401 with the body Jwt is not in the form of Header.Payload.Signature for an API key sent as Authorization: Bearer means the platform is older than 1.2.1 — use the X-Api-Key header there, or upgrade

Unexpected 404 from a model endpoint

The internal inference gateway forwards only allow-listed paths, so a healthy model answers 404 on any other path. Confirm the model is listed in /v1/models, and that the request uses an exact path from the inference endpoint allow-list rather than a sub-path of one.

Probe endpoints are not exposed through the gateway, so check the workload from inside the pod:

kubectl get pods -n <model-namespace>
kubectl logs <model-pod> -n <model-namespace>
kubectl exec -n <model-namespace> <model-pod> -- curl -sS localhost:8000/health

A 404 on /health, /ping, or /version through the gateway is expected, not an outage. When an application needs a path that is not served, add it to allowed_paths instead of disabling the allow-list.

When To Escalate

Escalate beyond first-line troubleshooting when:

• multiple platform layers fail at once
• Argo CD, ingress, and observability all show inconsistent state
• the issue appears related to cluster infrastructure rather than platform configuration
• the problem is reproducible across environments with the same manifests

Related Pages

Hybrid search reports a missing BM25 field

The collection uses an older schema. Dense mode can keep existing semantic searches working, but enabling Hybrid or rerunning a same-model reindex does not add the missing field. Follow collection compatibility and recovery to preserve source data and rebuild the index.