Share GPUs with time slicing and fractioning

Run several models on one GPU by using NVIDIA time slicing, KAI Scheduler GPU fractioning, or both on different node pools of the same cluster.

Agentic Friendly

The platform can share one physical GPU between several model workloads in two ways:

  • Time slicing: the NVIDIA device plugin advertises one GPU as several nvidia.com/gpu.shared replicas. All replicas see the full GPU memory, and the GPU switches between them in time. There is no memory isolation.
  • Fractioning: KAI Scheduler gives a pod a fraction of a GPU (for example 0.25) through the gpu-fraction annotation. The KAI gpu-fractioning components enforce a memory limit for each fraction on the node. This limit is not a security or fault-isolation boundary. Fractioning requires NVIDIA driver major version 615 or later on the GPU nodes.

You can enable both modes on the same cluster. Each GPU node pool uses one mode, which you select with a node label. Whole-GPU profiles keep working on GPU nodes without a sharing mode and on fractioning nodes. A time-sliced node offers only shared replicas, so whole-GPU pods do not run there.

How the modes are separated

MechanismTime slicingFractioning
Node label nvidia.com/device-plugin.configtimeslicingfractioning
Device-plugin config on the nodeGPU split into replicas × nvidia.com/gpu.sharedGPU stays whole (nvidia.com/gpu)
Who splits the GPUNVIDIA device pluginKAI Scheduler
GPU memoryNo limit; vLLM limits itself to its shareLimit per fraction, enforced by KAI gpu-fractioning
Isolation boundaryNoneNone; pods share one physical GPU
Minimum NVIDIA driverAny driver supported by the GPU OperatorMajor version 615 or later

A time-sliced node advertises nvidia.com/gpu: 0, so KAI cannot place a fractional pod on it. Each shared runtime profile also has a required node affinity on the matching label value.

Shared GPUs are not an isolation boundary

Neither mode isolates workloads from each other. Workloads on the same GPU share one physical device and can affect each other's stability and performance. With fractioning, a fatal GPU fault in one workload also stops the other workloads on that GPU. Do not put workloads that need strict isolation on a shared pool. Use whole-GPU profiles on dedicated GPUs, or validated MIG partitions, for them.

Prerequisites

  • The platform GPU Operator is installed (bsqai helm-apply gpu-operator). See Enable GPU for AI components.
  • [dra] enabled = false in the gpu-operator component. Time slicing needs the device plugin, and DRA mode removes it. The render fails if you enable both.
  • For fractioning: GPU nodes with NVIDIA driver major version 615 or later. If the provider image has an older driver, let the GPU Operator install the driver (see Driver ownership).
  • For fractioning: containerd 2.0 or later with NRI enabled on the GPU nodes, and Kubernetes 1.28 or later.
  • The KAI gpu-fractioning images (gpu-fractioning/operator, fractiond, metricsd, mpsd) are mirrored into the platform registry.

Driver ownership

The gpu-operator [driver] managed setting applies to the whole cluster. When it is true, the GPU Operator installs the driver and the NVIDIA container toolkit on the GPU nodes. On a node that already has a host driver, the GPU Operator detects that driver and does not install its own, but it still deploys the container toolkit. Set it to true only when the cluster has at least one GPU node pool without a host driver.

Host-driver nodes must opt out of the toolkit

When managed = true, nodes that already have a host driver and container runtime (for example AKS node pools created with the default GPU driver) must opt out with the node label nvidia.com/gpu.deploy.container-toolkit=false. Without this label, the toolkit changes the host containerd configuration and containerd does not start. On AKS, node auto-repair then reboots the node in a loop.

Node poolDriver source[driver] managedRequired node label
All GPU pools have a host driver (≥ 615 for fractioning)Provider imagefalsenone
Pool without a host driver (on AKS, created with --gpu-driver none)GPU Operatortruenone
Pool with a host driver, in a cluster with managed = trueProvider imagetruenvidia.com/gpu.deploy.container-toolkit=false

To keep one driver source per cluster, create all GPU node pools without a host driver when managed = true.

Configure the platform

Both modes are off by default. Turn them on in platform.env, as for every site setting; do not edit platform.toml:

GPU_SHARING_TIME_SLICING_ENABLED=true
# Integer from 2 to 16
GPU_SHARING_TIME_SLICING_REPLICAS=4

GPU_SHARING_FRACTIONING_ENABLED=true
# JSON list of unique values, each at least 0.06 and less than 1; it replaces the whole list
GPU_SHARING_FRACTIONING_FRACTIONS='[0.25,0.5]'

If a fractioning pool needs the operator-installed driver, also set:

GPU_OPERATOR_DRIVER_MANAGED=true
OverrideDefaultKeyEffect
GPU_SHARING_TIME_SLICING_ENABLED, GPU_SHARING_TIME_SLICING_REPLICASfalse, 4[gpu_sharing.time_slicing] enabled, replicas in platform.tomlAdds the timeslicing device-plugin config and the -timesliced runtime profiles
GPU_SHARING_FRACTIONING_ENABLED, GPU_SHARING_FRACTIONING_FRACTIONSfalse, [0.25, 0.5][gpu_sharing.fractioning] enabled, fractions in platform.tomlSets KAI gpuSharingMode: NvFractions, deploys gpu-fractioning, adds one -fraction-<value> runtime profile per fraction
GPU_OPERATOR_DRIVER_MANAGEDfalse[driver] managed in gpu-operator/variables.tomlGPU Operator installs the driver and container toolkit
GPU_OPERATOR_IMAGES_DRIVER615.71.09[images] driver in gpu-operator/variables.tomlDriver version that the GPU Operator installs (must be ≥ 615 for fractioning)
GPU_OPERATOR_DRA_ENABLEDfalse[dra] enabled in gpu-operator/variables.tomlMust stay false

The render stops when a value is out of range or when time slicing is on with GPU_OPERATOR_DRA_ENABLED=true.

Prepare the node pools

Set the labels and taints on the node pool, not on single nodes, so that they persist when nodes are replaced, reimaged, or scaled out. The platform's GPU workloads tolerate any value of the nvidia.com/gpu taint; present matches the other GPU pages.

Label or taintTime-slicing poolFractioning pool
workloadgpugpu
nvidia.com/device-plugin.configtimeslicingfractioning
nvidia.com/gpu.deploy.container-toolkitfalse on a host-driver pool when [driver] managed = true; otherwise not setfalse on a host-driver pool when [driver] managed = true; otherwise not set
Taintnvidia.com/gpu=present:NoSchedulenvidia.com/gpu=present:NoSchedule

Example for AKS:

# Time slicing on the AKS-supplied driver
az aks nodepool add --resource-group <rg> --cluster-name <cluster> --name gputs \
  --node-vm-size <gpu-sku> \
  --labels workload=gpu nvidia.com/device-plugin.config=timeslicing nvidia.com/gpu.deploy.container-toolkit=false \
  --node-taints nvidia.com/gpu=present:NoSchedule

# Fractioning on the operator-installed driver
az aks nodepool add --resource-group <rg> --cluster-name <cluster> --name gpufrac \
  --node-vm-size <gpu-sku> --gpu-driver none \
  --labels workload=gpu nvidia.com/device-plugin.config=fractioning \
  --node-taints nvidia.com/gpu=present:NoSchedule

Apply the change

bsqai apply all -r
bsqai push-manifests "Enable GPU sharing"
bsqai helm-apply gpu-operator

The GPU Operator is not deployed through GitOps, so you must run bsqai helm-apply gpu-operator again after you change [driver] managed, [gpu_sharing.time_slicing], or the driver image.

Select a profile for a model

Each mode adds LLMInferenceServiceConfig profiles next to the whole-GPU profiles. Every GPU profile, shared or whole, uses schedulerName: kai-scheduler and the RuntimeClass named by GPU_OPERATOR_RUNTIME_CLASS, nvidia-container-runtime by default. See NVIDIA RuntimeClass.

Time-sliced profileFractional profile
Profile namebsqai-config-llm-gpu-timeslicedbsqai-config-llm-gpu-fraction-025, bsqai-config-llm-gpu-fraction-05, …
GPU requestnvidia.com/gpu.shared: 1none; annotation gpu-fraction: "<value>"
vLLM GPU memory utilization0.9 / replicas0.80, as on a whole GPU; KAI limits the pod to its fraction's memory
Node affinitynvidia.com/device-plugin.config=timeslicingnvidia.com/device-plugin.config=fractioning
Label bsqai.io/gpu-allocation-modetimeslicedfractional

The S3-streaming profile family (bsqai-config-llm-gpu-s3-streaming) gets the same variants.

Allow the shared profiles for tenants

A tenant deploys only on the resource profiles its deployment policy allows. What a new policy allows depends on the tenant:

TenantResource profiles allowed when the tenant is first set up
Global Organization, on a fresh installationThe CPU profile bsqai-config-llm-cpu, the whole-GPU profiles, and the time-sliced and fractional profiles that the installation renders
Every other tenantOnly the whole-GPU profiles bsqai-config-llm-gpu, -gpu-2, -gpu-4 and -gpu-8, so its models never share a GPU

No policy allows the S3-streaming profiles, including their shared variants, or the GPU profiles that platform administrators define at run time, until a platform administrator adds them. Platform administrators define those profiles in Define GPU profiles. The base profile bsqai-config-llm-gpu-shared is never offered for deployment.

To let a tenant use shared GPUs, a platform administrator opens Administration → Tenants, selects the tenant, opens Deployment policy, and adds the shared profiles to its resource profiles. The Global Organization needs the same step when GPU sharing is enabled after installation, or after an upgrade from an earlier release: an existing policy keeps the profiles it already has.

Shared profiles carry no tenant affinity

One shared GPU can hold models of different tenants. Keep a tenant that must not share hardware on whole-GPU profiles.

Set the tenant's GPU allocation

A tenant's models are also bound by its GPU allocation. A platform administrator sets it in the same tenant, on the Quotas tab, in the GPU allocation card:

FieldEffect
Guaranteed GPUsGPUs the tenant's models are guaranteed across all its deployments. Empty or 0 guarantees nothing.
GPU limitGPUs the tenant's models may use together. Empty means no cap. It must be at least the guarantee.

Both accept fractions of one GPU, such as 0.25 or 0.5, with at most three decimals. A shared profile counts for its share of the card: 1 / replicas for a time-sliced profile, and the fraction for a fractional profile. The allocation applies once inference is enabled for the tenant.

A guarantee above 0 needs GPUs that the platform reserves for tenants with KAI_SCHEDULER_EXTRAS_TENANT_GPU_QUOTA; see Tenant GPU guarantees. For how the tenant's GPU queues enforce the allocation, see Tenant GPU scheduling. A pod on a shared GPU is metered for its share of the card; see Manage monthly BCU allowances and caps.

To check where shared models run and which tenants hold each GPU, a platform administrator opens Administration → Platform → GPU usage; see Review GPU usage.

Verify

# Node labels per pool
kubectl get nodes -l workload=gpu \
  -L nvidia.com/device-plugin.config,nvidia.com/gpu.deploy.driver,nvidia.com/gpu.deploy.container-toolkit

# Time-slicing nodes advertise shared replicas
kubectl get nodes -l nvidia.com/device-plugin.config=timeslicing \
  -o custom-columns=NAME:.metadata.name,SHARED:.status.allocatable.nvidia\\.com/gpu\\.shared

# Fractioning nodes report a supported driver
kubectl get nodes -l nvidia.com/device-plugin.config=fractioning -L nvidia.com/cuda.driver-version.major

Troubleshooting

SymptomCauseAction
GPU node goes NotReady about one minute after each boot, with ContainerRuntimeIsDown and repeated rebootsContainer toolkit deployed on a node that has a host driverSet nvidia.com/gpu.deploy.container-toolkit=false on the node pool, run bsqai helm-apply gpu-operator again, and reimage the node if containerd still fails
Node condition gpu-fractioning.nvidia.com/Ready=False, reason GPUDriverVersionMissingDriver older than 615 on a node that runs KAI gpu-fractioningUse a driver ≥ 615 on fractioning pools: a host driver, or the operator-installed driver when the image has an older one; on time-slicing pools this condition does not block scheduling
fractiond or mpsd pods in ImagePullBackOffgpu-fractioning images not in the platform registryMirror the images for the KAI Scheduler chart version in use
Time-sliced model runs out of GPU memoryReplicas do not have memory isolationLower replicas, or use a fractional profile
A deployment on a shared profile fails with deployment_not_allowedThe tenant's deployment policy does not allow the profileAdd the profile to the tenant's deployment policy (see Allow the shared profiles for tenants)

On this page