KServe
Model-serving control plane for KServe LLM and predictive inference workloads.
Component Category
AI serving / model inference
Component Description
KServe is the Kubernetes-native model-serving framework used by BullSequana AI. The platform installs KServe 0.21.0 and its LLM inference APIs, uses LLMInferenceService resources for LLM deployments, and deploys supported registered MLflow model versions as predictive InferenceService resources, on an MLServer-compatible runtime or, for ONNX models, on NVIDIA Triton Inference Server.
Why It Is Used
In BullSequana AI, KServe provides declarative model lifecycle, autoscaling, runtime configuration, workload status, and an internal inference data path.
Learn More
Deployment notes
KServe's controllers run in the kserve namespace, two replicas each, under the separate inference Argo CD parent application. The layer includes KServe CRDs and resources, LLM inference CRDs, KAI Scheduler, Envoy AI Gateway, the Gateway API Inference Extension, runtime profiles, and a private internal gateway.
The Global organization's models run in the kserve namespace. A customer organization's models run in its tenant namespace, tenant-<slug>, once a platform administrator enables inference for the tenant. KAI Scheduler places their pods in the tenant's GPU queue, within the GPU guarantee and cap that a platform administrator sets. The namespace admits only the serving runtimes and resource profiles that the tenant's deployment policy allows; without a policy, it admits no model deployment.
The shipped vLLM 0.32.0 profiles include CPU and NVIDIA GPU configurations for 1, 2, 4, and 8 GPUs. Optional profiles stream Safetensors from the platform object store or from a dedicated [model_storage] S3 endpoint. When the site enables GPU sharing, a time-sliced profile, bsqai-config-llm-gpu-timesliced, and fractional profiles, such as bsqai-config-llm-gpu-fraction-025 and bsqai-config-llm-gpu-fraction-05, are rendered too; both modes are off by default. Platform administrators can add their own GPU profiles on top of the whole-GPU base bsqai-config-llm-gpu and the shared base bsqai-config-llm-gpu-shared. Their names cannot start with the built-in prefixes bsqai-config- or kserve-config-.
Administrators curate the deployment choices exposed by the guided model setup through Portal model presets.
Every model request passes through the BSQAI API, which sends it through the model's route to the private KServe inference gateway. LLM requests use the OpenAI-compatible API. Predictive models use KServe's V2 inference protocol, MLflow's scoring format, or a custom container's HTTP API, and applications call them by serving endpoint. A predictive model's pods accept traffic only from the internal gateway and the platform's metrics collectors.
The platform's bsqai-mlserver-mlflow3 serving runtime serves models logged with MLflow 3 and MLflow 2, for amd64 and arm64 nodes. It answers a caller's bad input with 400, takes traffic only once its model is loaded, and stops taking traffic before it shuts down. With at least two replicas, the default for predictive models, restarts and new versions therefore drop no requests. A model with one replica is unavailable while its only pod restarts. The upstream kserve-mlserver runtime remains available for models logged with MLflow 1 or 2. See Deploy a predictive model.
The bsqai-triton and bsqai-triton-gpu runtimes serve ONNX models logged with MLflow on NVIDIA Triton Inference Server 26.09, on CPU or on one GPU, for amd64 and arm64 nodes. They use NVIDIA's own image, mirrored to the platform registry. When the container starts, it arranges the registered model's files into the layout Triton reads, so nothing is downloaded at startup. The deployment's batch limit becomes the model's max_batch_size, with dynamic batching on. Like the MLflow 3 runtime, they take traffic only once the model is loaded, stop taking traffic before they shut down, and run two replicas by default.
Interacts With
BSQAI API, which creates deployments, routes authenticated LLM requests to them, and exposes workload health, pod, and log operations.MLflowandMLServer, which provide registered model versions and compatible predictive serving, including the platform's MLflow 3 runtime.NVIDIA Triton Inference Server, which serves ONNX models on CPU or GPU.vLLM, which serves LLM workloads through platform runtime profiles.KAI Scheduler, which places GPU workloads and enforces each tenant's GPU queue.BSQAI Tenant Operator, which prepares each tenant serving namespace, its GPU queue, and its deployment-policy admission.Envoy AI Gateway, which provides the internal model-serving data path.Rook-Cephor external S3-compatible storage, which supplies model artifacts.Grafana AlloyandGrafana, which collect and display inference telemetry.