Enable GPU for AI components

Turn on GPU acceleration for speech-to-text (Speaches/Whisper) and Docling using environment-variable overrides.

Two BSQAI API subcharts ship with GPU acceleration that is off by default: speech-to-text through Speaches and Docling workers. KServe model workloads use separate runtime profiles described under Inference.

GPU is controlled by a per-component boolean in variables.toml. Flipping it through the standard environment-variable override switches the container image and adds nvidia.com/gpu resource requests, runtimeClassName: nvidia, and GPU node affinity and tolerations.

You set these overrides in platform.env (shared, committed) or local.env (personal, gitignored). See Configuration model for how the two files are layered.

Prerequisites

The cluster must be able to schedule GPU pods. GPU nodes must provide:

node label workload=gpu
node taint nvidia.com/gpu=present
a compatible NVIDIA driver and container runtime
the nvidia RuntimeClass and NVIDIA device plugin

If the cluster owner does not already provide the Kubernetes-level NVIDIA integration, install the optional platform-supplied GPU Operator after validating cluster access:

bsqai helm-apply gpu-operator

The platform values enable node feature discovery, the device plugin, GPU feature discovery, DCGM metrics, and MIG support. Driver and container-toolkit installation are disabled, so those node-level prerequisites must already exist. Skip this command when the cluster owner supplies an equivalent supported integration.

For the full GPU footprint and sizing guidance, see Minimum requirements.

By default the platform uses preferred (soft) GPU affinity, so GPU pods prefer GPU nodes but can still schedule elsewhere. To make GPU placement a hard requirement, set:

# Pin GPU pods to gpu-labeled nodes ([common].enforce_affinity, default false)
COMMON_ENFORCE_AFFINITY="true"

Enable GPU

Add the toggles you need to platform.env (or local.env):

# Enable GPU for speech-to-text (Speaches/Whisper) in llm-backend
LLM_BACKEND_SPEACHES_GPU="true"

# Select the Docling CUDA image
LLM_BACKEND_DOCLING_GPU="true"

# Run one RQ worker in the GPU pool
LLM_BACKEND_DOCLING_SCALING_GPU_WORKER_REPLICAS="1"

Speech-to-text (Speaches/Whisper)


Component	`llm-backend`
Toggle	`[speaches].gpu` in `variables.toml`
Override variable	`LLM_BACKEND_SPEACHES_GPU`

When set to true:

image switches from the digest-pinned CPU artifact to the digest-pinned CUDA artifact
sets WHISPER__INFERENCE_DEVICE=cuda and WHISPER__COMPUTE_TYPE=float16
requests nvidia.com/gpu: "1"
selects the GPU Whisper model (deepdml/faster-whisper-large-v3-turbo-ct2) instead of the CPU model (Systran/faster-whisper-base), and propagates it to PIPECAT_WHISPER_MODEL

Docling


Component	`llm-backend` Docling subchart
Toggle	`[docling].gpu` and `[docling_scaling].gpu_worker_replicas`
Override variables	`LLM_BACKEND_DOCLING_GPU`, `LLM_BACKEND_DOCLING_SCALING_GPU_WORKER_REPLICAS`

When set to true:

image switches from docling-serve-cpu to docling-serve-cu128
adds nvidia.com/gpu: "1" to GPU worker requests and limits

Apply the change

After editing platform.env, re-render and push manifests so ArgoCD picks up the CUDA images and GPU resource requests:

uv run -m src.cli.main apply all -r
uv run -m src.cli.main push-manifests "Enable GPU for Speaches and Docling"

See Platform CLI for details on these commands.

Environment-variable override convention

The override variable names above follow the platform's standard convention, so any future GPU-capable component is self-explanatory:

COMPONENT_NAME + TOML_TABLE + KEY   (uppercased, hyphens replaced with underscores)

The component name is uppercased with - replaced by _ (llm-backend becomes LLM_BACKEND).
The TOML table and key are joined with _.
Nested table names remain in the key, which is why the worker count uses LLM_BACKEND_DOCLING_SCALING_GPU_WORKER_REPLICAS.

See Configuration model for the complete rules.