Enable GPU for AI components

Two CoreAI components ship with GPU acceleration that is off by default (CPU mode): speech-to-text (Speaches/Whisper) in llm-backend, and docling-server. This playbook explains how to enable GPU for each one.

GPU is controlled by a per-component boolean in variables.toml. Flipping it through the standard environment-variable override switches the container image and adds nvidia.com/gpu resource requests, runtimeClassName: nvidia, and GPU node affinity and tolerations.

You set these overrides in platform.env (shared, committed) or local.env (personal, gitignored). See Configuration model for how the two files are layered.

Prerequisites

The cluster must be able to schedule GPU pods. GPU nodes must provide:

• node label workload=gpu
• node taint nvidia.com/gpu=present
• the nvidia RuntimeClass, with NVIDIA drivers and the NVIDIA device plugin installed at cluster level

For the full GPU footprint and sizing guidance, see Minimum requirements.

By default the platform uses preferred (soft) GPU affinity, so GPU pods prefer GPU nodes but can still schedule elsewhere. To make GPU placement a hard requirement, set:

# Pin GPU pods to gpu-labeled nodes ([common].enforce_affinity, default false)
COMMON_ENFORCE_AFFINITY="true"

Enable GPU

Add the toggles you need to platform.env (or local.env):

# Enable GPU for speech-to-text (Speaches/Whisper) in llm-backend
LLM_BACKEND_SPEACHES_GPU="true"

# Enable GPU for docling-server
DOCLING_SERVER_COMPONENT_GPU="true"

Speech-to-text (Speaches/Whisper)

When set to true:

• image switches to speaches-ai/speaches:latest-cuda
• sets WHISPER__INFERENCE_DEVICE=cuda and WHISPER__COMPUTE_TYPE=float16
• requests nvidia.com/gpu: "1"
• selects the GPU Whisper model (deepdml/whisper-large-v3-turbo) instead of the CPU model (Systran/faster-whisper-base), and propagates it to PIPECAT_WHISPER_MODEL

Docling

When set to true:

• image switches from docling-serve-cpu to docling-serve-cu128
• adds nvidia.com/gpu: "1" to both requests and limits

Apply the change

After editing platform.env, re-render and push manifests so ArgoCD picks up the CUDA images and GPU resource requests:

uv run -m src.cli.main apply all -r
uv run -m src.cli.main push-manifests "Enable GPU for Speaches and Docling"

See Platform CLI for details on these commands.

Environment-variable override convention

The override variable names above follow the platform's standard convention, so any future GPU-capable component is self-explanatory:

COMPONENT_NAME + TOML_TABLE + KEY   (uppercased, hyphens replaced with underscores)

• The component name is uppercased with - replaced by _ (llm-backend → LLM_BACKEND, docling-server → DOCLING_SERVER).
• The TOML table and key are joined with _.
• Keys that live under the [component] table include the COMPONENT_ segment — that is why Docling uses DOCLING_SERVER_COMPONENT_GPU while Speaches (under [speaches]) uses LLM_BACKEND_SPEACHES_GPU.

See the variable resolution chain for the complete rules.

Related pages

• Configuration model
• Minimum requirements
• Platform CLI
• Docling component