Inference
KServe model serving, private gateways, runtime profiles, and GPU-aware scheduling.
BullSequana AI separates model serving into an inference Argo CD parent application. Higher-level AI services choose and expose models; this layer operates their Kubernetes workloads.
Main inference components
| Component | Main role |
|---|---|
| KServe | Reconciles LLMInferenceService and predictive InferenceService workloads |
| vLLM | Executes high-throughput language-model inference |
| Speaches serving runtimes | ClusterServingRuntime resources, bsqai-speaches and bsqai-speaches-gpu, that serve speech models registered in MLflow as KServe InferenceService workloads |
| KAI Scheduler | Places and quotas GPU workloads |
| Internal model gateway | ClusterIP Envoy Gateway, kserve-internal, that admits only namespaces labeled for model serving |
| Envoy AI Gateway | Provides the private inference data path |
| Gateway API Inference Extension | Adds model-aware routing to the gateway path |
Runtime profiles
The platform supplies CPU and NVIDIA GPU vLLM profiles for 1, 2, 4, and 8 GPUs. A profile defines the image, resources, model-loading behavior, and runtime arguments for a KServe deployment. Operators can tune or add profiles for site hardware. To run several models on one GPU, see Share GPUs with time slicing and fractioning. Fractioning requires NVIDIA driver 615 or later.
Platform administrators can also define GPU profiles at run time, in the GPU profiles section of Administration → Tenant → Allowed Resource Profiles. A GPU profile takes a share of one GPU, an amount of GPU memory on one GPU, or one or more whole GPUs on one node. It can be pinned to one GPU model and can reserve its own CPU and memory. These profiles build on the platform's bsqai-config-llm-gpu and bsqai-config-llm-gpu-shared profiles, and a tenant can use one only when its deployment policy allows it. The BSQAI API exposes the same operations under /v1/resource-profiles/gpu-profiles. See Define GPU profiles.
GPU profiles reserve CPU and memory per GPU and cap their model container at the same amount: 2 CPU and 16 GiB per GPU by default, or that share for a shared-GPU profile. The CPU profile reserves 6 CPU and caps it there. Every LLM model, served with a vLLM profile, also runs a router-scheduler pod that reserves and is capped at 1 CPU and 2 GiB. A site changes these with KSERVE_RUNTIME_CONFIGS_GPU_CPU_PER_GPU, KSERVE_RUNTIME_CONFIGS_GPU_MEMORY_PER_GPU_GIB, KSERVE_RUNTIME_CONFIGS_ROUTER_CPU, and KSERVE_RUNTIME_CONFIGS_ROUTER_MEMORY. Lower the per-GPU values when your GPU nodes have less than 2 CPU and 16 GiB free per GPU, or GPU models stay Pending.
GPU profiles schedule only onto nodes labeled nvidia.com/gpu.present=true, which the NVIDIA GPU Operator sets. If a GPU model pod stays Pending, check which nodes carry the label:
kubectl get nodes -L nvidia.com/gpu.presentThe bsqai-triton-gpu runtime for ONNX models requests one GPU. It is placed on the same nvidia.com/gpu.present=true nodes and tolerates the nvidia.com/gpu taint. A site whose GPU nodes carry another label or taint sets KSERVE_RUNTIME_CONFIGS_TRITON_GPU_PLACEMENT_NODE_LABEL and _NODE_VALUE, and _TOLERATIONS as JSON, for example [{"key":"sku","operator":"Equal","value":"gpu","effect":"NoSchedule"}] for a pool tainted sku=gpu. Its pods are non-preemptible, so KAI Scheduler starts them only within the queue's guaranteed GPUs. For a Global Organization model, that guarantee is KAI_SCHEDULER_EXTRAS_GPU_QUOTA on the default queue, and it is 0 by default. For a tenant model, it is the tenant's Guaranteed GPUs. With no guarantee, a Triton GPU model stays Pending with NonPreemptibleOverQuota. The GPUs that platform administrators can guarantee to tenants are held in the parent queue through KAI_SCHEDULER_EXTRAS_TENANT_GPU_QUOTA, also 0 by default.
GPU profiles also select the NVIDIA RuntimeClass named by the GPU_OPERATOR_RUNTIME_CLASS platform setting. If that RuntimeClass does not exist on the cluster, Kubernetes does not create the model pod. See Enable GPU for AI components.
Model artifacts can come from Hugging Face, a PVC, the MLflow model registry, the platform object store, a tenant's own bucket, or an optional dedicated [model_storage] S3 endpoint. Direct-S3 profiles stream Safetensors from approved S3-compatible storage. See Supported model locations.
Where models run
Models of the Global Organization run in the kserve namespace. A tenant's own models run in its tenant namespace, tenant-<slug>, once a platform administrator enables inference for the tenant. The tenant operator then labels the namespace for model serving, which admits it to the internal model gateway, and creates the tenant's KAI Scheduler queues. Speech models on the Speaches runtimes are deployed by the Global Organization only.
Cluster admission holds every tenant model to the tenant's deployment policy. A tenant model may use only the serving runtimes and resource profiles the policy allows, runs a platform serving runtime pod without custom containers, volumes, or host settings, and takes its resources from its resource profile. A tenant model's GPU pods must be scheduled by KAI Scheduler in the tenant's inference queue. Only the platform can set or change the namespace's serving label and the policy it records on the namespace.
The tenant's guaranteed GPUs and GPU limit come from its GPU allocation. See Tenant GPU scheduling.
Request path
Model workloads are private. Applications authenticate to the BSQAI API and call its OpenAI-compatible endpoints, such as /v1/chat/completions and /v1/responses. The API resolves the model's route and sends each call for an in-cluster model through the internal inference gateway to KServe. Workloads are not intended to have their own public route. A route to an external provider connection leaves the cluster from the BSQAI API and does not use the internal gateway.
The gateway forwards a request only when its path is on an explicit allow-list, matched exactly under the /<namespace>/<model> prefix. Anything else has no backend and answers 404: sub-paths such as /v1/chat/completions/render, and model-server control, tokenizer, and probe endpoints such as /health, /version, and /v1/load_lora_adapter. Kubelet probes reach the pod directly and are unaffected.
The default list is the OpenAI-compatible inference surface: /v1/completions, /v1/chat/completions, /v1/responses, /v1/messages, /v1/models, /v1/embeddings, /v1/score, /v1/rerank, /v1/audio/transcriptions, and /v1/audio/translations.
The allow-list limits which paths are exposed, not who calls them. Callers are authenticated and authorized earlier, at the BSQAI API.
A NetworkPolicy limits which pods reach the internal gateway at all. It admits, by name, only the BSQAI API workloads that call models: the API itself and its file-rag-worker, reindex-worker, deep-research-worker, docint, and docint-orchestration-worker workloads, in the BSQAI API's namespace. Prometheus, and the kubelet probe address ranges a site configures, reach only the gateway's metrics and readiness port. Any other pod's request to the gateway is dropped. The policy takes effect only on a cluster whose network plugin enforces NetworkPolicy.
Predictive models, served as KServe InferenceService resources, are reached on the same internal gateway by their hostname, <name>-<namespace>.<gateway host>, rather than by path. The BSQAI API sends only the operations that its serving endpoints offer, and their pods accept traffic only from the internal gateway, Prometheus, and Alloy. Sites that configure kubelet probe address ranges admit those too.
Inference endpoint allow-list
Operators control the allow-list through the kserve-runtime-configs component. Its [route_authorization] table applies to every CPU, GPU, and direct-S3 runtime profile.
| Key | Default | Effect |
|---|---|---|
enabled | true | Applies the allow-list. false restores the permissive upstream KServe router preset. |
allowed_paths | the ten OpenAI-compatible inference paths listed in Request path | Every path reachable through the gateway, under the /<namespace>/<model> prefix. |
core_paths | empty | Subset of allowed_paths also reachable under the /publishers/<namespace>/models/<model> prefix and through the model-based-routing header. |
Both path settings are comma-separated strings, so each one is a single environment override:
KSERVE_RUNTIME_CONFIGS_ROUTE_AUTHORIZATION_ENABLED=true
KSERVE_RUNTIME_CONFIGS_ROUTE_AUTHORIZATION_ALLOWED_PATHS="/v1/chat/completions,/v1/models"
KSERVE_RUNTIME_CONFIGS_ROUTE_AUTHORIZATION_CORE_PATHS=""The route CRDs cap a route at 16 rules: one per allowed_paths entry, one per core_paths entry, and one shared model-routing rule when core_paths is set. The shipped default uses 10.
Rendering fails with an explicit message for an invalid path, a duplicate rule name, a core_paths entry missing from allowed_paths, or a list over the rule budget. Add a required path to allowed_paths rather than widening the match type.
See Configuration model for how component tables and environment overrides are resolved.
Disabling the allow-list re-exposes the model server
enabled = false restores catch-all routing to the model workload, so its unauthenticated control, tokenizer, and adapter endpoints become reachable through the internal gateway again. Use it only as a temporary rollback while an allow-list problem is corrected.
Operational boundary
Foundation owns the serving control plane, scheduling, networking, storage integration, and telemetry. AI owns model registration, tenant authorization, model discovery, and the stable application APIs.