# KAI Scheduler (/docs/foundation/components/kai-scheduler)



## Component Category [#component-category]

Cluster operations / GPU scheduling

## Component Description [#component-description]

KAI Scheduler is a Kubernetes scheduler for AI workloads. It places pods into a hierarchy of queues, each with a guaranteed share of resources, an optional cap, and a priority, and it can give a pod a fraction of a GPU. In BullSequana AI it schedules the GPU model pods of the Global Organization and of every tenant.

## Why It Is Used [#why-it-is-used]

In BullSequana AI, KAI Scheduler turns a tenant's GPU allocation into an enforced scheduling boundary. One tenant's models cannot take GPUs guaranteed to another tenant, and a tenant's cap bounds all of its GPU work together. The same scheduler places fractional GPU pods when GPU sharing is enabled.

## Learn More [#learn-more]

* [KAI Scheduler documentation](https://github.com/NVIDIA/KAI-Scheduler/tree/main/docs)
* [NVIDIA/KAI-Scheduler on GitHub](https://github.com/NVIDIA/KAI-Scheduler)

## Queues [#queues]

KAI Scheduler places pods only in leaf queues. The platform builds this hierarchy:

<Mermaid
  chart="flowchart TD
    P[&#x22;default-parent-queue: GPU quota = platform GPU quota + tenant GPU quota&#x22;] --> D[&#x22;default-queue: Global Organization models in kserve&#x22;]
    P --> T[&#x22;tenant-tenantId: tenant guarantee and cap&#x22;]
    T --> TI[&#x22;tenant-tenantId-inference: model serving, priority 100&#x22;]
    T --> TT[&#x22;tenant-tenantId-training: pods with priorityClassName train, priority 50&#x22;]"
/>

| Queue                                  | Created by                                                                                                                             | GPU quota                                                                                          |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `default-parent-queue`                 | The `kai-scheduler` component                                                                                                          | `KAI_SCHEDULER_EXTRAS_GPU_QUOTA` plus `KAI_SCHEDULER_EXTRAS_TENANT_GPU_QUOTA`, both `0` by default |
| `default-queue`                        | The `kai-scheduler` component                                                                                                          | `KAI_SCHEDULER_EXTRAS_GPU_QUOTA`                                                                   |
| `tenant-<tenantId>` and its two leaves | The [BSQAI Tenant Operator](/docs/foundation/components/tenant-operator#tenant-gpu-scheduling), while the tenant has inference enabled | The tenant's **Guaranteed GPUs**                                                                   |

Every queue may use idle GPUs beyond its quota up to its limit. The default queues have no limit, and a tenant's parent queue is capped at the tenant's **GPU limit**. The BSQAI API refuses a tenant guarantee that the parent queue's tenant GPU quota cannot hold. See [Tenant GPU guarantees](/docs/deployment/configuration-model#tenant-gpu-guarantees).

GPU model pods run with the `build-preemptible` priority class (`[gpu] priority_class` of the `kserve-runtime-configs` component). KAI Scheduler starts a preemptible pod above its queue's quota when GPUs are free. A non-preemptible pod, such as a Triton GPU model, starts only within the guaranteed GPUs and otherwise stays `Pending` with `NonPreemptibleOverQuota`.

## Tenant admission [#tenant-admission]

The `tenant-gpu-queue` ValidatingAdmissionPolicy, shipped with this component, checks every GPU pod in a tenant namespace. A pod counts as a GPU pod when it requests `nvidia.com/gpu` or carries the `gpu-fraction` or `gpu-memory` annotation. The pod is admitted only when the namespace is admitted for serving, it uses `schedulerName: kai-scheduler` without `spec.nodeName`, and its `kai.scheduler/queue` label names one of the tenant's two leaves. Only pods with `priorityClassName: train` may use the training leaf. The policy fails closed: an error in its evaluation refuses the pod.

## GPU sharing [#gpu-sharing]

With `[gpu_sharing.fractioning] enabled = true` in `platform.toml`, KAI Scheduler runs in `NvFractions` sharing mode and deploys its `gpu-fractioning` components. It then places pods that carry a `gpu-fraction` annotation on a fraction of one GPU and limits each fraction's GPU memory on the node. Fractions are not an isolation boundary. Time slicing is handled by the NVIDIA device plugin instead, and KAI Scheduler places those pods as ordinary pods on shared replicas. See [Share GPUs with time slicing and fractioning](/docs/deployment/playbooks/gpu-sharing).

## Deployment notes [#deployment-notes]

KAI Scheduler v0.18 deploys in the `kai-scheduler` namespace at sync wave 3 in the inference tier. It renders its configuration through Argo CD instead of a Helm hook job, and its images come from the platform registry. Its queue controller and scheduler expose metrics to the platform Prometheus. The platform's GPU attribution rules read the queues' allocated, guaranteed, and fair-share GPUs from them for the **Administration** > **GPU usage** page; see [Review GPU usage](/docs/administration/gpu-usage).

## Interacts With [#interacts-with]

* `KServe` and `vLLM`, whose GPU model pods KAI Scheduler places.
* `BSQAI Tenant Operator`, which creates each tenant's queues and removes them when inference is disabled.
* `BSQAI API`, which labels tenant model deployments with the tenant's inference queue and checks tenant guarantees against the parent queue.
* `NVIDIA GPU Operator`, whose device plugin advertises the GPUs and time-sliced replicas that KAI Scheduler allocates.
* `Prometheus`, which scrapes the queue metrics behind the GPU usage views.
