vLLM
High-throughput LLM inference engine in the Foundation layer.
Component Category
Inference / LLM serving engine
Component Description
vLLM is a high-throughput and memory-efficient inference engine designed for serving large language models.
Why It Is Used
In BullSequana AI Foundation, vLLM powers efficient LLM inference with strong performance characteristics for production workloads, especially where throughput and GPU utilization matter.
Learn More
Deployment notes
vLLM is not deployed as a standalone platform service. KServe creates vLLM workloads from LLMInferenceService resources and operator-managed runtime profiles. The platform ships vLLM 0.32.0 images for GPU and for CPU, with profiles for CPU and 1, 2, 4, or 8 NVIDIA GPUs. Each model also runs an llm-d router endpoint picker, which schedules requests across the model's replicas.
With GPU sharing enabled, more profiles run a model on part of a GPU: bsqai-config-llm-gpu-timesliced on time-sliced nodes, and bsqai-config-llm-gpu-fraction-025, bsqai-config-llm-gpu-fraction-05, and any other configured fraction on fractioning nodes. See Runtime profiles and Share GPUs with time slicing and fractioning. A tenant uses only the runtimes and resource profiles its deployment policy allows.
Some profiles can stream Safetensors directly from S3-compatible model storage.
Inference remains private behind the BSQAI API and the internal inference gateway. Applications reach models through the BSQAI API's OpenAI-compatible endpoints, such as POST /v1/chat/completions and POST /v1/responses. The internal gateway forwards only the allowed paths to vLLM: completions, chat completions, responses, messages, models, embeddings, score, rerank, and audio transcriptions and translations. Any other path returns 404.
The Portal prefills vLLM deployments from the vLLM recipe catalogue when it covers the model being deployed.
Interacts With
KServe, which creates and manages vLLM model-serving workloads.BSQAI APIand Studio, which deploy, change, and remove vLLM-backed models through the model installer endpoints (/v1/model-installer/...) and route inference requests to them.KAI Scheduler, which places GPU model pods and enforces tenant GPU queues.