Share GPUs with time slicing and fractioning
Run several models on one GPU by using NVIDIA time slicing, KAI Scheduler GPU fractioning, or both on different node pools of the same cluster.
The platform can share one physical GPU between several model workloads in two ways:
- Time slicing: the NVIDIA device plugin advertises one GPU as several
nvidia.com/gpu.sharedreplicas. All replicas see the full GPU memory, and the GPU switches between them in time. There is no memory isolation. - Fractioning: KAI Scheduler gives a pod a fraction of a GPU (for example
0.25) through thegpu-fractionannotation. The KAIgpu-fractioningcomponents enforce a memory limit for each fraction on the node. This limit is not a security or fault-isolation boundary. Fractioning requires NVIDIA driver major version 615 or later on the GPU nodes.
You can enable both modes on the same cluster. Each GPU node pool uses one mode, which you select with a node label. Whole-GPU profiles keep working on GPU nodes without a sharing mode and on fractioning nodes. A time-sliced node offers only shared replicas, so whole-GPU pods do not run there.
How the modes are separated
| Mechanism | Time slicing | Fractioning |
|---|---|---|
Node label nvidia.com/device-plugin.config | timeslicing | fractioning |
| Device-plugin config on the node | GPU split into replicas × nvidia.com/gpu.shared | GPU stays whole (nvidia.com/gpu) |
| Who splits the GPU | NVIDIA device plugin | KAI Scheduler |
| GPU memory | No limit; vLLM limits itself to its share | Limit per fraction, enforced by KAI gpu-fractioning |
| Isolation boundary | None | None; pods share one physical GPU |
| Minimum NVIDIA driver | Any driver supported by the GPU Operator | Major version 615 or later |
A time-sliced node advertises nvidia.com/gpu: 0, so KAI cannot place a fractional pod on it. Each shared runtime profile also has a required node affinity on the matching label value.
Shared GPUs are not an isolation boundary
Neither mode isolates workloads from each other. Workloads on the same GPU share one physical device and can affect each other's stability and performance. With fractioning, a fatal GPU fault in one workload also stops the other workloads on that GPU. Do not put workloads that need strict isolation on a shared pool. Use whole-GPU profiles on dedicated GPUs, or validated MIG partitions, for them.
Prerequisites
- The platform GPU Operator is installed (
bsqai helm-apply gpu-operator). See Enable GPU for AI components. [dra] enabled = falsein thegpu-operatorcomponent. Time slicing needs the device plugin, and DRA mode removes it. The render fails if you enable both.- For fractioning: GPU nodes with NVIDIA driver major version 615 or later. If the provider image has an older driver, let the GPU Operator install the driver (see Driver ownership).
- For fractioning: containerd 2.0 or later with NRI enabled on the GPU nodes, and Kubernetes 1.28 or later.
- The KAI
gpu-fractioningimages (gpu-fractioning/operator,fractiond,metricsd,mpsd) are mirrored into the platform registry.
Driver ownership
The gpu-operator [driver] managed setting applies to the whole cluster. When it is true, the GPU Operator installs the driver and the NVIDIA container toolkit on the GPU nodes. On a node that already has a host driver, the GPU Operator detects that driver and does not install its own, but it still deploys the container toolkit. Set it to true only when the cluster has at least one GPU node pool without a host driver.
Host-driver nodes must opt out of the toolkit
When managed = true, nodes that already have a host driver and container runtime (for example AKS node pools created with the default GPU driver) must opt out with the node label nvidia.com/gpu.deploy.container-toolkit=false. Without this label, the toolkit changes the host containerd configuration and containerd does not start. On AKS, node auto-repair then reboots the node in a loop.
| Node pool | Driver source | [driver] managed | Required node label |
|---|---|---|---|
| All GPU pools have a host driver (≥ 615 for fractioning) | Provider image | false | none |
Pool without a host driver (on AKS, created with --gpu-driver none) | GPU Operator | true | none |
Pool with a host driver, in a cluster with managed = true | Provider image | true | nvidia.com/gpu.deploy.container-toolkit=false |
To keep one driver source per cluster, create all GPU node pools without a host driver when managed = true.
Configure the platform
Both modes are off by default. Turn them on in platform.env, as for every site setting; do not edit platform.toml:
GPU_SHARING_TIME_SLICING_ENABLED=true
# Integer from 2 to 16
GPU_SHARING_TIME_SLICING_REPLICAS=4
GPU_SHARING_FRACTIONING_ENABLED=true
# JSON list of unique values, each at least 0.06 and less than 1; it replaces the whole list
GPU_SHARING_FRACTIONING_FRACTIONS='[0.25,0.5]'If a fractioning pool needs the operator-installed driver, also set:
GPU_OPERATOR_DRIVER_MANAGED=true| Override | Default | Key | Effect |
|---|---|---|---|
GPU_SHARING_TIME_SLICING_ENABLED, GPU_SHARING_TIME_SLICING_REPLICAS | false, 4 | [gpu_sharing.time_slicing] enabled, replicas in platform.toml | Adds the timeslicing device-plugin config and the -timesliced runtime profiles |
GPU_SHARING_FRACTIONING_ENABLED, GPU_SHARING_FRACTIONING_FRACTIONS | false, [0.25, 0.5] | [gpu_sharing.fractioning] enabled, fractions in platform.toml | Sets KAI gpuSharingMode: NvFractions, deploys gpu-fractioning, adds one -fraction-<value> runtime profile per fraction |
GPU_OPERATOR_DRIVER_MANAGED | false | [driver] managed in gpu-operator/variables.toml | GPU Operator installs the driver and container toolkit |
GPU_OPERATOR_IMAGES_DRIVER | 615.71.09 | [images] driver in gpu-operator/variables.toml | Driver version that the GPU Operator installs (must be ≥ 615 for fractioning) |
GPU_OPERATOR_DRA_ENABLED | false | [dra] enabled in gpu-operator/variables.toml | Must stay false |
The render stops when a value is out of range or when time slicing is on with GPU_OPERATOR_DRA_ENABLED=true.
Prepare the node pools
Set the labels and taints on the node pool, not on single nodes, so that they persist when nodes are replaced, reimaged, or scaled out. The platform's GPU workloads tolerate any value of the nvidia.com/gpu taint; present matches the other GPU pages.
| Label or taint | Time-slicing pool | Fractioning pool |
|---|---|---|
workload | gpu | gpu |
nvidia.com/device-plugin.config | timeslicing | fractioning |
nvidia.com/gpu.deploy.container-toolkit | false on a host-driver pool when [driver] managed = true; otherwise not set | false on a host-driver pool when [driver] managed = true; otherwise not set |
| Taint | nvidia.com/gpu=present:NoSchedule | nvidia.com/gpu=present:NoSchedule |
Example for AKS:
# Time slicing on the AKS-supplied driver
az aks nodepool add --resource-group <rg> --cluster-name <cluster> --name gputs \
--node-vm-size <gpu-sku> \
--labels workload=gpu nvidia.com/device-plugin.config=timeslicing nvidia.com/gpu.deploy.container-toolkit=false \
--node-taints nvidia.com/gpu=present:NoSchedule
# Fractioning on the operator-installed driver
az aks nodepool add --resource-group <rg> --cluster-name <cluster> --name gpufrac \
--node-vm-size <gpu-sku> --gpu-driver none \
--labels workload=gpu nvidia.com/device-plugin.config=fractioning \
--node-taints nvidia.com/gpu=present:NoScheduleApply the change
bsqai apply all -r
bsqai push-manifests "Enable GPU sharing"
bsqai helm-apply gpu-operatorThe GPU Operator is not deployed through GitOps, so you must run bsqai helm-apply gpu-operator again after you change [driver] managed, [gpu_sharing.time_slicing], or the driver image.
Select a profile for a model
Each mode adds LLMInferenceServiceConfig profiles next to the whole-GPU profiles. Every GPU profile, shared or whole, uses schedulerName: kai-scheduler and the RuntimeClass named by GPU_OPERATOR_RUNTIME_CLASS, nvidia-container-runtime by default. See NVIDIA RuntimeClass.
| Time-sliced profile | Fractional profile | |
|---|---|---|
| Profile name | bsqai-config-llm-gpu-timesliced | bsqai-config-llm-gpu-fraction-025, bsqai-config-llm-gpu-fraction-05, … |
| GPU request | nvidia.com/gpu.shared: 1 | none; annotation gpu-fraction: "<value>" |
| vLLM GPU memory utilization | 0.9 / replicas | 0.80, as on a whole GPU; KAI limits the pod to its fraction's memory |
| Node affinity | nvidia.com/device-plugin.config=timeslicing | nvidia.com/device-plugin.config=fractioning |
Label bsqai.io/gpu-allocation-mode | timesliced | fractional |
The S3-streaming profile family (bsqai-config-llm-gpu-s3-streaming) gets the same variants.
Allow the shared profiles for tenants
A tenant deploys only on the resource profiles its deployment policy allows. What a new policy allows depends on the tenant:
| Tenant | Resource profiles allowed when the tenant is first set up |
|---|---|
| Global Organization, on a fresh installation | The CPU profile bsqai-config-llm-cpu, the whole-GPU profiles, and the time-sliced and fractional profiles that the installation renders |
| Every other tenant | Only the whole-GPU profiles bsqai-config-llm-gpu, -gpu-2, -gpu-4 and -gpu-8, so its models never share a GPU |
No policy allows the S3-streaming profiles, including their shared variants, or the GPU profiles that platform administrators define at run time, until a platform administrator adds them. Platform administrators define those profiles in Define GPU profiles. The base profile bsqai-config-llm-gpu-shared is never offered for deployment.
To let a tenant use shared GPUs, a platform administrator opens Administration → Tenants, selects the tenant, opens Deployment policy, and adds the shared profiles to its resource profiles. The Global Organization needs the same step when GPU sharing is enabled after installation, or after an upgrade from an earlier release: an existing policy keeps the profiles it already has.
Shared profiles carry no tenant affinity
One shared GPU can hold models of different tenants. Keep a tenant that must not share hardware on whole-GPU profiles.
Set the tenant's GPU allocation
A tenant's models are also bound by its GPU allocation. A platform administrator sets it in the same tenant, on the Quotas tab, in the GPU allocation card:
| Field | Effect |
|---|---|
| Guaranteed GPUs | GPUs the tenant's models are guaranteed across all its deployments. Empty or 0 guarantees nothing. |
| GPU limit | GPUs the tenant's models may use together. Empty means no cap. It must be at least the guarantee. |
Both accept fractions of one GPU, such as 0.25 or 0.5, with at most three decimals. A shared profile counts for its share of the card: 1 / replicas for a time-sliced profile, and the fraction for a fractional profile. The allocation applies once inference is enabled for the tenant.
A guarantee above 0 needs GPUs that the platform reserves for tenants with KAI_SCHEDULER_EXTRAS_TENANT_GPU_QUOTA; see Tenant GPU guarantees. For how the tenant's GPU queues enforce the allocation, see Tenant GPU scheduling. A pod on a shared GPU is metered for its share of the card; see Manage monthly BCU allowances and caps.
To check where shared models run and which tenants hold each GPU, a platform administrator opens Administration → Platform → GPU usage; see Review GPU usage.
Verify
# Node labels per pool
kubectl get nodes -l workload=gpu \
-L nvidia.com/device-plugin.config,nvidia.com/gpu.deploy.driver,nvidia.com/gpu.deploy.container-toolkit
# Time-slicing nodes advertise shared replicas
kubectl get nodes -l nvidia.com/device-plugin.config=timeslicing \
-o custom-columns=NAME:.metadata.name,SHARED:.status.allocatable.nvidia\\.com/gpu\\.shared
# Fractioning nodes report a supported driver
kubectl get nodes -l nvidia.com/device-plugin.config=fractioning -L nvidia.com/cuda.driver-version.majorTroubleshooting
| Symptom | Cause | Action |
|---|---|---|
GPU node goes NotReady about one minute after each boot, with ContainerRuntimeIsDown and repeated reboots | Container toolkit deployed on a node that has a host driver | Set nvidia.com/gpu.deploy.container-toolkit=false on the node pool, run bsqai helm-apply gpu-operator again, and reimage the node if containerd still fails |
Node condition gpu-fractioning.nvidia.com/Ready=False, reason GPUDriverVersionMissing | Driver older than 615 on a node that runs KAI gpu-fractioning | Use a driver ≥ 615 on fractioning pools: a host driver, or the operator-installed driver when the image has an older one; on time-slicing pools this condition does not block scheduling |
fractiond or mpsd pods in ImagePullBackOff | gpu-fractioning images not in the platform registry | Mirror the images for the KAI Scheduler chart version in use |
| Time-sliced model runs out of GPU memory | Replicas do not have memory isolation | Lower replicas, or use a fractional profile |
A deployment on a shared profile fails with deployment_not_allowed | The tenant's deployment policy does not allow the profile | Add the profile to the tenant's deployment policy (see Allow the shared profiles for tenants) |