KAI Scheduler

Kubernetes GPU scheduler that places model pods into hierarchical queues with GPU guarantees, caps, and sharing.

Agentic Friendly

Component Category

Cluster operations / GPU scheduling

Component Description

KAI Scheduler is a Kubernetes scheduler for AI workloads. It places pods into a hierarchy of queues, each with a guaranteed share of resources, an optional cap, and a priority, and it can give a pod a fraction of a GPU. In BullSequana AI it schedules the GPU model pods of the Global Organization and of every tenant.

Why It Is Used

In BullSequana AI, KAI Scheduler turns a tenant's GPU allocation into an enforced scheduling boundary. One tenant's models cannot take GPUs guaranteed to another tenant, and a tenant's cap bounds all of its GPU work together. The same scheduler places fractional GPU pods when GPU sharing is enabled.

Learn More

Queues

KAI Scheduler places pods only in leaf queues. The platform builds this hierarchy:

QueueCreated byGPU quota
default-parent-queueThe kai-scheduler componentKAI_SCHEDULER_EXTRAS_GPU_QUOTA plus KAI_SCHEDULER_EXTRAS_TENANT_GPU_QUOTA, both 0 by default
default-queueThe kai-scheduler componentKAI_SCHEDULER_EXTRAS_GPU_QUOTA
tenant-<tenantId> and its two leavesThe BSQAI Tenant Operator, while the tenant has inference enabledThe tenant's Guaranteed GPUs

Every queue may use idle GPUs beyond its quota up to its limit. The default queues have no limit, and a tenant's parent queue is capped at the tenant's GPU limit. The BSQAI API refuses a tenant guarantee that the parent queue's tenant GPU quota cannot hold. See Tenant GPU guarantees.

GPU model pods run with the build-preemptible priority class ([gpu] priority_class of the kserve-runtime-configs component). KAI Scheduler starts a preemptible pod above its queue's quota when GPUs are free. A non-preemptible pod, such as a Triton GPU model, starts only within the guaranteed GPUs and otherwise stays Pending with NonPreemptibleOverQuota.

Tenant admission

The tenant-gpu-queue ValidatingAdmissionPolicy, shipped with this component, checks every GPU pod in a tenant namespace. A pod counts as a GPU pod when it requests nvidia.com/gpu or carries the gpu-fraction or gpu-memory annotation. The pod is admitted only when the namespace is admitted for serving, it uses schedulerName: kai-scheduler without spec.nodeName, and its kai.scheduler/queue label names one of the tenant's two leaves. Only pods with priorityClassName: train may use the training leaf. The policy fails closed: an error in its evaluation refuses the pod.

GPU sharing

With [gpu_sharing.fractioning] enabled = true in platform.toml, KAI Scheduler runs in NvFractions sharing mode and deploys its gpu-fractioning components. It then places pods that carry a gpu-fraction annotation on a fraction of one GPU and limits each fraction's GPU memory on the node. Fractions are not an isolation boundary. Time slicing is handled by the NVIDIA device plugin instead, and KAI Scheduler places those pods as ordinary pods on shared replicas. See Share GPUs with time slicing and fractioning.

Deployment notes

KAI Scheduler v0.18 deploys in the kai-scheduler namespace at sync wave 3 in the inference tier. It renders its configuration through Argo CD instead of a Helm hook job, and its images come from the platform registry. Its queue controller and scheduler expose metrics to the platform Prometheus. The platform's GPU attribution rules read the queues' allocated, guaranteed, and fair-share GPUs from them for the Administration > GPU usage page; see Review GPU usage.

Interacts With

  • KServe and vLLM, whose GPU model pods KAI Scheduler places.
  • BSQAI Tenant Operator, which creates each tenant's queues and removes them when inference is disabled.
  • BSQAI API, which labels tenant model deployments with the tenant's inference queue and checks tenant guarantees against the parent queue.
  • NVIDIA GPU Operator, whose device plugin advertises the GPUs and time-sliced replicas that KAI Scheduler allocates.
  • Prometheus, which scrapes the queue metrics behind the GPU usage views.

On this page