Model Installer API
Register model sources and deploy them as KServe LLM or predictive inference services.
The Model Installer API turns an approved model location into a platform-managed inference deployment. It is part of the BSQAI API and is available under /v1/model-installer.
What registration does
POST /v1/model-installer/register_model:
- validates the source, the runtime settings, and the organization's deployment policy;
- creates or updates a KServe
LLMInferenceService, or anInferenceServicefor a predictive model, in the serving namespace of the active organization; and - adds or updates the model's route, so that the BSQAI API serves the model under its
name.
The call returns once the KServe resource is written, before the model serves. Poll the health route until its status is ready.
Applications should use the BSQAI API rather than call the private KServe endpoint directly.
Organizations and namespaces
The active organization decides where a model is deployed, which MLflow it is read from, and who may operate it:
| Global organization | Customer organization | |
|---|---|---|
| Namespace | kserve | tenant-<slug>, the organization's own |
| Model name callers send | name | <slug>/<name>, for example acme/support-chat |
| Registry and imports | The Global MLflow | The organization's private MLflow, with artifacts under s3://<slug>-data/mlflow/ |
| Permissions | platform.global_service.manage; platform.global.read for the runtime routes | The inference.* permissions of the Inference Deployer and Inference Operator roles |
| Who may call the model | Allow rules in each organization that the model's audience admits: set by platform administrators in the Global organization, and by tenant administrators in their own | Its owner, the rules given in callers, then the rules that its owner or its Grant Managers set. Once it is published, each other tenant's administrators set their own tenant's rules |
The namespace field of a request, and the {namespace} segment of the unregister path, are kept for existing clients and ignored. A route that names a deployment, such as the adapter update or the health route, accepts the bare name or, in a customer organization, <slug>/<name>.
A customer organization deploys only when a platform administrator has enabled inference for it. Otherwise the deploy, import, deployment, download, and recipe routes, and the runtime routes, answer 409 model_installer_tenant_inference_disabled. Imports and the deployment of a registered version also need the organization's private MLflow to be ready.
Deployment policy
Each organization deploys and imports within the deployment policy that platform administrators set for it: the models, serving runtimes, and resource profiles it may use. A model is matched by its registered model name or its Hugging Face repository ID, and a model from any other location, such as s3://, by its full URL. The policy applies to deployments and to imports (download_hf_model, download_s3_model, register_s3_model), and an import that also deploys is checked again before it deploys.
| Status | Code | Cause |
|---|---|---|
| 400 | deployment_not_allowed | The model, runtime, or resource profile named in param is outside the policy. The message lists the allowed values. |
| 409 | deployment_no_resource_profile | The organization has no resource profile enabled, so nothing can be deployed. |
Existing deployments keep running when the policy changes. GET /v1/model-installer/deployments reports outside_policy: true for each deployment the current policy no longer allows.
Storage of a customer organization
In a customer organization, an s3:// location, for the model, for an adapter, or for register_s3_model, must lie in the organization's own bucket, s3://<slug>-data/. Any other bucket answers 400 model_installer_source_outside_tenant before anything is deployed. hf:// and mlflow-artifacts:/ locations are accepted. download_s3_model, which copies a model from any configured S3 storage, and delete_model are available in the Global organization only, and answer 503 mlflow_global_only in a customer organization.
Ownership and initial access
In a customer organization, the caller who deploys a model becomes its owner. callers lists the allow rules the model starts with:
"callers": [
{ "subject_type": "group", "subject_id": "<group-uuid>", "relation": "invoker" },
{ "subject_type": "user", "subject_id": "<user-uuid>", "relation": "invoker" }
]subject_type is user, group, or all_members; subject_id names the user or group and is omitted for All Members. relation is invoker, which lets the subject call the model; viewer is refused with 400 serving_endpoint_viewer_rule_retired, because every member already sees the organization's models. A user or group outside the organization, and a rule that would change nothing, are refused before anything is deployed: the deployer themselves, a rule named twice, or other rules beside all_members (409 serving_endpoint_grant_changes_nothing). callers has no default: with an empty list, only the owner can call the model until they add a rule. In the Global organization, callers is ignored.
The adapters of a deployment get the base model's owner and rules whenever their adapters change. Undeploying removes the model's owner and rules before it removes the deployment, including the rules that other organizations set on a model offered to them, so a later deployment under the same name starts without any.
A customer organization's model serves only that organization until its tenant administrators publish it. To change the rules, publish the model to other tenants, or manage its Owner and Grant Managers afterwards, see Model access and publishing API, or Model access in Studio.
Predictive models
A predictive model from the MLflow registry is served as a KServe InferenceService. Registration adds its route too, in the protocol its serving runtime declares: KServe's V2 inference protocol for MLServer and Triton, or a custom container's own HTTP API. Applications call it through the BSQAI API by its serving endpoint ID, as Use predictive models via API describes.
- A predictive model deploys with 2 replicas, so a restart of one pod leaves the other serving. Set
minReplicasto choose another count; a lowermaxReplicasalso caps the default. With one replica, the model is unavailable while its pod restarts. - A model logged with MLflow's
onnxflavor can also run on NVIDIA Triton, onbsqai-triton(CPU) orbsqai-triton-gpu(one GPU).max_batch_size, from 1 to 1,024, sets the largest batch Triton accepts in one request. Without it, Triton chooses its own limit, 4 for a model whose batch dimension is variable. Any other engine refuses the field with 400model_installer_max_batch_size_unsupported. - A serving runtime that declares no protocol cannot serve a callable model. Registration on it fails with 400
model_installer_engine_route_protocol_missing, before anything is deployed. - Speech models are the exception: they join the platform's shared Speaches instance without a route, and speech-to-text and text-to-speech reach them through their own backend settings. Only the Global organization deploys them. A customer organization gets 400
model_installer_speaches_deployment_invalid.
The model's route
The route that registration adds:
- serves the model under its name,
namein the Global organization or<slug>/<name>in a customer organization, and asks the model server forserved_model_name, which defaults toname; - reaches the model through a connection named
kserve/<namespace>/<name>, with the request'stimeout(600 seconds by default) andmax_retries(5 by default); - takes its mode from
model_mode, its limits frommax_tokens,max_input_tokens, andmax_output_tokens, and its tags fromtags, which default to["on-prem"]; and - states only the capabilities that the vLLM flags of a chat or completion deployment prove.
--reasoning-parsermakes reasoning Yes,--enable-auto-tool-choicewith--tool-call-parsermakes tool calling Yes, and--language-model-onlymakes image input No. All other capabilities stay Not stated, because the runtime profile can add flags of its own.
Each LoRA adapter that the deployment serves gets a route of its own on the same connection, as Update adapters describes.
Administrators can correct the capabilities of an installed chat model under Model routing in Studio. The corrections stay until the model is registered again with other settings. Removing the deployment also removes its route, its adapters' routes, and its connection.
Registration fails with 409 model_route_alias_taken when another route already uses the model's name. The installer then removes the model's deployment, even one that was running before the request. See Cloud and external models for the routes that administrators add.
Supported model locations
The url field accepts these source schemes:
| Scheme | Use |
|---|---|
hf://<organization>/<model> | Download model artifacts from Hugging Face |
pvc://<claim>/<optional-path> | Use artifacts already stored on a persistent volume |
s3://<bucket>/<path> | Read artifacts from configured S3-compatible model storage. A customer organization reads only its own bucket. |
Other URL schemes are rejected by the current registration API.
Runtime selection
Use runtime_config_name to select a platform-managed LLMInferenceServiceConfig, a resource profile. BullSequana AI ships:
- the CPU profile
bsqai-config-llm-cpu; - the whole-GPU profiles
bsqai-config-llm-gpufor one NVIDIA GPU, andbsqai-config-llm-gpu-2,-gpu-4, and-gpu-8, which shard the model over their GPUs with tensor parallelism; - with GPU sharing enabled, the shared profiles
bsqai-config-llm-gpu-timeslicedandbsqai-config-llm-gpu-fraction-<share>, such asbsqai-config-llm-gpu-fraction-05for half a GPU; and - where the site enables it, the
bsqai-config-llm-gpu-s3-streamingfamily, which streams Safetensors directly from the platform object store or from the optional[model_storage]endpoint.
Platform administrators can define further GPU profiles: a fraction of one GPU, a share of one GPU's memory in GiB, or several whole GPUs, optionally pinned to a GPU model and with their own CPU and memory reservation. In the Portal, they do this in Administration, on the Allowed Resource Profiles page, with New GPU profile under GPU profiles. Through the API, GET /v1/resource-profiles/gpu-models lists the GPU models the cluster's nodes report, and GET and POST /v1/resource-profiles/gpu-profiles and DELETE /v1/resource-profiles/gpu-profiles/{name} list, define, and delete these profiles. A profile that a deployment policy still allows cannot be deleted, and models already running on a deleted profile keep running.
An organization deploys only on the profiles its deployment policy allows. GET /v1/resource-profiles lists them for the active organization.
The legacy resourceProfile field remains available as a compatibility mapping for existing clients. New integrations should use runtime_config_name.
targetRequests controls the target request concurrency used by KServe autoscaling. The platform operator determines which profiles and limits are available at a site.
Register a model
The exact optional fields depend on the selected engine and runtime profile. A typical vLLM request is:
{
"name": "mistral-small",
"engine": "VLLM",
"features": ["TextGeneration"],
"url": "hf://mistralai/Mistral-Small-3.2-24B-Instruct-2506",
"runtime_config_name": "bsqai-config-llm-gpu",
"minReplicas": 0,
"maxReplicas": 2,
"targetRequests": 1,
"model_mode": "chat",
"max_input_tokens": 8192,
"max_output_tokens": 2048
}Use a profile that your organization's deployment policy allows.
Arguments for the model server go in args, each as an object with a name and an optional value, or as a string such as "--max-model-len=32768". The installer trims spaces around the name and value of each argument given as an object. Arguments given as strings are passed as they are.
A predictive model names a registered model version and a serving runtime instead of a URL:
{
"name": "churn-model",
"engine": "bsqai-mlserver-mlflow3",
"features": [],
"registry_model_name": "churn",
"registry_model_version": "3"
}An ONNX model on Triton can set its batch limit:
{
"name": "image-classifier",
"engine": "bsqai-triton-gpu",
"features": [],
"registry_model_name": "mobilenet",
"registry_model_version": "1",
"max_batch_size": 16
}GET /v1/inference-engines lists the serving runtimes and the model formats each accepts. bsqai-mlserver-mlflow3 serves models logged with MLflow 3 and MLflow 2. kserve-mlserver serves models logged with MLflow 1 or 2. bsqai-triton and bsqai-triton-gpu declare the mlflow-onnx format and serve models logged with MLflow's onnx flavor; an MLflow model directory with its single .onnx file. Registering the same name again with another version replaces the model in place, without dropping requests, and keeps its serving endpoint ID.
max_tokens, max_input_tokens, and max_output_tokens are positive integers, and registration sets no upper bound on them. The Portal deployment form accepts values up to 10,000,000. A model preset accepts up to 1,000,000, which the preset API enforces.
For an embedding model, pass max_input_tokens at registration. It sets where the BSQAI API splits long texts before it embeds them, as Capabilities describes. Without it, texts are split at 2,048 tokens, which a model with a shorter input limit, such as a 512-token model, rejects. Administrators cannot change an installed model's limits in Studio. To set or change them, register the model again with the new values. The built-in bge-m3 and Qwen3 embedding presets fill in their models' input limits, 8,192 and 32,768 tokens.
Tool calling
A vLLM model calls tools when it is served with enable-auto-tool-choice and the tool-call-parser of its model family. Chat sends tools to a chat model unless its route states tool calling No. vLLM refuses tools from a model served without tool calling: the UI chat stream then ends with an error event whose code is model_tools_unsupported, and the HTTP status stays 200. For a model whose route states tool calling No, chat refuses a request that needs tools before the stream starts, with 400 model_tools_unsupported.
The installer adds both arguments to a vLLM model that generates text when the request names neither of them and its tags do not hold vllm-recipe, the tag of a deployment from a vLLM recipe. A model generates text when model_mode is chat or completion, or when model_mode is omitted and features holds TextGeneration. The installer reads the model's family from model_type in its Hugging Face config.json:
| Family | model_type | tool-call-parser |
|---|---|---|
| Qwen | qwen2, qwen2_moe, qwen3, qwen3_moe, qwen3_next, qwen3_vl, qwen3_vl_moe | hermes |
| Mistral | mistral, mistral3, mixtral | mistral |
| gpt-oss | gpt_oss | openai |
For an hf://<organization>/<model> URL, the installer reads the config from Hugging Face with the installation's Hugging Face token. For an s3:// URL in the organization's model storage, such as a registered version's, it reads the config through the organization's MLflow. A pvc:// URL, or an s3:// URL outside that storage, has no config that the installer reads. For a chat or completion model, the route then states tool calling Yes, as The model's route describes.
For any other model type, or when the config cannot be read, the installer deploys the model without tool calling and says so in the response's message:
{
"status": "success",
"message": "Model llama-3-1-8b created successfully. Tool calling is off: no tool-call parser is known for the model type 'llama'. Add the vLLM arguments tool-call-parser and enable-auto-tool-choice to turn it on."
}Qwen3 Coder is published as qwen3_moe, as the other Qwen3 mixture-of-experts models are, so the installer gives it hermes. Give it qwen3_coder in args, and give a model of another family the parser that vLLM documents for it. The installer leaves a request's own tool-call arguments as they are:
"args": [
{ "name": "enable-auto-tool-choice" },
{ "name": "tool-call-parser", "value": "qwen3_coder" }
]A model that is already running keeps its arguments until it is registered again.
Deploy with LoRA adapters
A vLLM deployment on an LLMInferenceService can serve LoRA adapters next to its base model. List them in adapters, and optionally set vLLM's LoRA limits:
| Field | Values | Effect |
|---|---|---|
adapters | A list of { "name": "<adapter-name>", "url": "<adapter-location>" } | The adapters the model server loads at startup |
loraMaxRank | 1 to 512 | The highest adapter rank the server accepts, vLLM's --max-lora-rank. vLLM's default is 16. |
loraMaxAdapters | 1 to 64 | How many adapters the server holds on the GPU at once, vLLM's --max-loras. vLLM's default is 1. |
An adapter name has up to 63 lowercase letters, digits, and hyphens. An adapter location starts with hf://, s3://, or mlflow-artifacts:/. The installer resolves an mlflow-artifacts:/ location to its s3:// path in the artifact bucket, as it does for the model's own URL. Any other location, pvc:// included, answers 422. The two limits apply only when the request lists at least one adapter.
{
"name": "mistral-small",
"engine": "VLLM",
"features": ["TextGeneration"],
"url": "hf://mistralai/Mistral-Small-3.2-24B-Instruct-2506",
"runtime_config_name": "bsqai-config-llm-gpu",
"model_mode": "chat",
"adapters": [
{ "name": "support-tone", "url": "s3://<bucket>/adapters/support-tone" }
],
"loraMaxRank": 64,
"loraMaxAdapters": 2
}Registration checks the adapters before it deploys anything. These checks are the same as for Update adapters:
- the adapter name rules, the location allowlist, and the limit ranges above, which answer 422;
- a name that repeats, matches the model's own name, or belongs to another model route, which answers 400
model_installer_adapter_name_conflict; and - a runtime profile that cannot load adapters, which answers 400
model_installer_runtime_config_without_adapters.
Registration sets no limit on the number of adapters. The 32-adapter limit and the 409 model_installer_adapters_changed conflict apply only to an update. Registration adds the adapters' routes as Adapter routes describes. If an adapter's route cannot be added, registration removes the deployment again.
Update adapters
PUT /v1/model-installer/deployments/{name}/adapters replaces the LoRA adapters of a running deployment of the active organization. It answers with the deployment as GET /v1/model-installer/deployments reports it.
{
"adapters": [
{ "name": "support-tone", "url": "s3://<bucket>/adapters/support-tone" },
{ "name": "contract-summary", "url": "hf://<organization>/<adapter>" }
],
"lora_max_rank": 64,
"lora_max_adapters": 2
}- The body is the complete set. An adapter that the body leaves out is removed, with its route. An empty
adapterslist removes every adapter and the LoRA limits, and the model server starts without LoRA support. - A request lists up to 32 adapters, with the same names and locations as at registration.
lora_max_rankis 1 to 512 andlora_max_adaptersis 1 to 64. This body uses snake_case field names, unlikeloraMaxRankandloraMaxAdaptersinregister_model. A value out of range answers 422. - The body replaces the whole LoRA block. A limit that the body leaves out returns to vLLM's default, 16 for
lora_max_rankand 1 forlora_max_adapters.
The route requires platform.global_service.manage in the Global organization, or inference.deployment.update in a customer organization.
Concurrent updates
The installer applies an update only if the deployment has not changed since the installer read it. When another update, or any other change to the deployment, lands first, the call answers 409 model_installer_adapters_changed and changes nothing. Read the current set with GET /v1/model-installer/deployments, then send the update again. The request carries no version field or header.
The adapter routes always follow the adapter set that KServe holds after the update, so the last update to land decides both the adapters and their routes.
Restart and downtime
KServe loads adapters only when the model server starts. Every adapter update restarts the deployment's pod. The base model and all its adapters are unavailable until the base model has loaded again. The call returns when the change is applied, not when the model serves again. Use the health route to see when the deployment is ready.
Compatible runtime profiles
KServe's storage initializer downloads the adapter files before the model server starts. A runtime profile that turns the storage initializer off, such as a profile that streams Safetensors from object storage, cannot load adapters. A deployment or update with adapters on such a profile is refused with 400 model_installer_runtime_config_without_adapters. Choose another profile.
Only an LLMInferenceService carries adapters. An update for a predictive model, served as a KServe InferenceService, is refused with 400 model_installer_adapters_unsupported.
Adapter routes
Each adapter gets a model route of its own:
- the adapter's name is both the model name that clients send and the upstream model that vLLM serves;
- the route uses the base deployment's connection,
kserve/<namespace>/<name>; - it copies the base route's mode, API, limits, capabilities, and tags, and adds the tags
loraandlora-base:<deployment-name>; and - the Model Installer manages it, like the base route, and
/v1/modelslists it under the adapter's name.
Registration, a later registration of the same deployment, and an adapter update all bring the adapter routes in line with the adapter set. They remove the deployment's adapter routes that the set no longer names, then add or refresh one route for each adapter in it. Removing the deployment removes its adapter routes with its own route and connection.
An adapter name must differ from the other adapters' names, from the deployment's name and served_model_name, and from every other model route's name. Otherwise the request is refused with 400 model_installer_adapter_name_conflict. An update that lists adapters also needs the deployment's own route. These checks run before KServe is changed, so a refused request leaves the deployment as it was.
Retry after a partial failure
The installer changes the deployment in KServe first, then its routes. When KServe accepts the new set but a route cannot be written, the call answers 500 model_installer_partial_deploy. The deployment already restarts with the new adapters. Send the same request again: adding a route that already exists refreshes it, so a repeated call ends with exactly one route for each adapter.
Errors
| Status | Code | Cause |
|---|---|---|
| 400 | model_installer_adapter_name_conflict | Two adapters share a name, an adapter has the base model's name, or another model route already has the name. |
| 400 | model_installer_runtime_config_without_adapters | The deployment's runtime profile turns off the storage initializer and cannot load adapters. |
| 400 | model_installer_adapters_unsupported | The deployment is a KServe InferenceService, not an LLMInferenceService. |
| 404 | model_installer_runtime_not_found | No deployment has that name in the organization's namespace. |
| 404 | model_installer_gateway_route_not_found | The request lists adapters, but the deployment has no route for them to share. |
| 409 | model_installer_adapters_changed | The deployment changed while the update was applied. Nothing was changed. Read the deployment again and repeat the request. |
| 422 | Validation error | A name, limit, or the number of adapters is out of range, or an adapter location does not start with hf://, s3://, or mlflow-artifacts:/. |
| 500 | model_installer_partial_deploy | KServe has the new set, but some adapter routes were not written. Repeat the request. |
| 400 | model_installer_source_outside_tenant | In a customer organization, an adapter's s3:// location is outside the organization's bucket. |
| 503 | model_installer_runtime_unavailable | The cluster cannot be reached or does not serve the KServe resource. |
List deployments
GET /v1/model-installer/deployments lists the models that the inference runtime holds in the active organization's namespace. For each deployment it reports:
route_id: the ID of the model route that calls the deployment, ornullwhen no route calls it, as for a speech model;adapters: the adapters that the deployment serves, each withnameandurl, or an empty list;lora_max_rank: the deployment's--max-lora-rank, ornullwhen vLLM's default applies; andlora_max_adapters: the deployment's--max-loras, ornullwhen vLLM's default applies.
route_id is the route's id in GET /v1/serving-endpoints and its model_id in GET /v1/models/info. It names the deployment's own route when the routing table holds it. Otherwise, for an LLMInferenceService, it names a route whose connection calls the deployment's in-cluster URL with no Host header, such as a route that an administrator added. Of several such routes, it names the one whose alias sorts first. It never names a LoRA adapter's route. PUT /v1/model-installer/deployments/{name}/adapters and GET /v1/mlflow/models/{model_name}/deployments report route_id in the same way.
An adapter's url is the location that the runtime loads, so an mlflow-artifacts:/ URI appears as its bucket path.
Repository imports
The same API includes asynchronous import and repository operations:
| Endpoint | Purpose |
|---|---|
POST /v1/model-installer/download_hf_model | Import a Hugging Face model into the platform repository |
POST /v1/model-installer/download_s3_model | Copy a model from S3-compatible storage into the repository. Global organization only. |
POST /v1/model-installer/register_s3_model | Register an existing S3 artifact in MLflow without copying it |
GET /v1/model-installer/downloads | List background downloads |
GET /v1/model-installer/downloads/{download_id}/status | Read download state |
GET /v1/model-installer/downloads/{download_id}/logs | Read download logs |
DELETE /v1/model-installer/delete_model | Delete a repository model. Global organization only. |
DELETE /v1/model-installer/unregister_model/{name}/{namespace} | Remove an inference deployment and route |
GET /v1/model-installer/downloads lists queued and running downloads. The status route reports a download as soon as the import request returns its download_id: queued until its background work starts, then running, then succeeded or failed. A finished download keeps its status and its logs for 24 hours; after that, the status route answers 404 and the logs are empty.
Imports write to the MLflow of the active organization, and list only that organization's downloads. Import first when the model needs a platform-managed MLflow artifact and version history. Register directly when the source URL is already the desired serving source.
Choose the experiment
Each import logs one MLflow run in an experiment. Name it with exactly one of:
mlflow_experiment_id: the numeric ID of an existing experiment.mlflow_experiment_name: an experiment name. A name nobody uses yet becomes a new experiment, owned by the requester.
download_hf_model uses the experiment named bsq-ai-models when neither is given; the S3 routes need one. For older clients, a non-numeric mlflow_experiment_id is read as a name. The experiment is checked before the import starts: an unknown ID, or an experiment the caller may not write, answers 404; a deleted experiment answers 409; both fields at once answer 422.
Recipe catalogue
A vLLM recipe is a published, verified configuration for serving one model with vLLM. The BSQAI API stores the catalogue in its own database and answers from that store, rather than reaching the upstream catalogue on each request.
| Endpoint | Purpose |
|---|---|
GET /v1/model-installer/recipes | List the models the catalogue covers |
GET /v1/model-installer/recipes/{hf_id} | Read one model's recipe, including its checkpoints and capabilities |
GET /v1/model-installer/recipes/status | Report model count, last sync, source, upstream availability, and the accepted snapshot size |
POST /v1/model-installer/recipes/refresh | Fetch the catalogue; an optional body reads from another address for this refresh only |
GET /v1/model-installer/recipes/snapshot | Export the whole catalogue as a file |
POST /v1/model-installer/recipes/import | Replace the whole catalogue with an exported file |
Each organization keeps its own catalogue. These endpoints require platform.global_service.manage in the Global organization, or inference.model.deploy in a customer organization. An import replaces the catalogue rather than merging into it, and the accepted file size is the one reported by recipes/status.
A refresh address must use https. The API refuses an address that resolves to a private network. Storing an address as the tenant default is a configuration change: it is written through PATCH /v1/config as model_catalog.vllm_recipes_base_url and requires tenant.update.
Installation settings
| Setting | Default | Effect |
|---|---|---|
VLLM_RECIPES_ENABLED | true | When false, the catalogue is not served and the Portal shows no recipe surfaces. |
VLLM_RECIPES_UPSTREAM_ENABLED | true | When false, nothing is fetched. The catalogue is filled by importing a snapshot. |
VLLM_RECIPES_BASE_URL | https://recipes.vllm.ai | The catalogue fetched by a refresh, unless a tenant address overrides it. |
HF_MODEL_CARD_ENABLED | true | When false, the Portal shows no Hugging Face card, badges, or links. |
MLFLOW_UI_URL | unset | Leave unset: MLflow is internal-only and has no UI address. When set, model, experiment, and run pages link to it. |
Air-gapped operation
An installation with VLLM_RECIPES_UPSTREAM_ENABLED=false fetches nothing. Export a snapshot from a connected installation, carry the file through the approved channel, and import it. See Air-gapped installations for the Portal procedure.
Portal flow
Studio's Deploy a model uses the same endpoints. Guided setup applies an operator-managed preset or a catalogue recipe; advanced setup exposes the model source, runtime and resource profile, scaling, timeouts, and model task and limits. Both flows deploy into the active organization's namespace and show the generated LLMInferenceService configuration before submission.
When the entered model is one the catalogue covers, the deployment form offers to prefill its runtime arguments, environment variables, features, and limits from that model's recipe.
Long-running imports appear under Downloads in progress on the Model repository page, with their progress, transfer speed, and time remaining. On the Download model page, a download that fails shows its last log lines. A failed import is not retried: start it again.
Authentication and tenant scope
In the Global organization, Model Installer endpoints require an authenticated user with platform.global_service.manage. In a customer organization, they require the matching inference.* permission: inference.model.deploy to deploy, import, and read the recipe catalogue and the Hugging Face card, inference.deployment.list to list deployments, inference.deployment.update to change adapters, and inference.deployment.delete to undeploy. MLflow registry operations have separate mlflow.model.* permissions, and the installer also checks the caller's access on the MLflow content it touches:
- Imports (
download_hf_model,download_s3_model,register_s3_model) need write on the target experiment when it already exists, and on the model when they add a version to an existing one. A model that does not exist yet is created when the request is accepted, owned by the requester; a failed import leaves it empty for a retry. - Deploying a registered version through
register_model, withregistry_model_nameandregistry_model_version, needs read on that model. A deployment given only a model URL is not checked against the registry. delete_modelchecks everything before deleting: an experiment needs its owner, a run needs write on its experiment, and a model needs its owner (one version needs write) plus write on every affected run's experiment. An emptied experiment the requester may not delete is kept.delete_modelanddownload_s3_modelwork with the Global MLflow only, and return 503mlflow_global_onlyin a customer organization. When a JWT belongs to more than one Keycloak Organization, includeX-Tenant-Id. API keys already carry their tenant binding.
Verify a deployment
Use the deployment routes of the Model Installer API to inspect the resulting workload. They require platform.global.read in the Global organization, or inference.deployment.list in a customer organization:
GET /v1/model-installer/deployments/{model_name}/health
GET /v1/model-installer/deployments/{model_name}/pods
GET /v1/model-installer/deployments/{model_name}/pods/{pod_name}/logs
GET /v1/model-installer/deployments/{model_name}/pods/{pod_name}/describeThe health response carries two checks. checks.gateway shows whether a model route has the model's name (registered), and whether that route's model server lists the model (responding). checks.kserve_runtime shows the state of the KServe resource. The status is ready when both checks pass, and downloading while the route exists but the runtime is not ready yet. For a predictive model, responding means that its server reports the model as loaded. A model without a route, such as a speech model, takes its status from checks.kserve_runtime alone.
When the organization's namespace holds no KServe resource with that name, for example for a model name that was never deployed, the health route answers 404 with the error code model_installer_runtime_not_found and param name. A 404 means that the deployment does not exist, not that it is still starting, so stop polling the route.
The status is failed as soon as one of the model's pods cannot start on its own, for example a model server that keeps crashing or an image that cannot be pulled. deployment says how far the deployment got:
| Field | Meaning |
|---|---|
deployment.stage | preparing, fetching_model, loading_model, or ready. ready means the model accepts requests. A runtime that streams the model from object storage goes from preparing straight to loading_model. |
deployment.reason | Why a deployment failed or is not moving, or null: waiting_for_resources, image_unavailable, configuration_error, model_fetch_failed, out_of_memory, server_crashed, not_responding, pods_refused, or status_unknown. waiting_for_resources means no node has room yet; the deployment continues when one does. pods_refused means the cluster refused to create the model's pods. |
deployment.refusal | With pods_refused, why: quota when the pods would exceed the namespace's resource quota or its per-container limits, policy when an admission policy refused them. Otherwise null. The cluster's own message is logged, not returned. |
deployment.uses_gpu | Whether the model's pods request a GPU, or null when no pod was read, as for a model that is already ready. |
The stage carries no percentage or time remaining, because KServe does not report reliable progress while it fetches or loads a model.
The routes take no namespace parameter. They read models and pods only in the namespace of the active organization, and {model_name} is the deployment name or, in a customer organization, <slug>/<name>. The BSQAI API changes KServe resources and reads pod logs only in kserve and in the serving namespaces of organizations with inference enabled. Its Kubernetes role for them is bound in each of those namespaces, never cluster-wide, and withdrawing an organization's inference removes it there. The logs route accepts the optional query parameters container and tail_lines (default 100).
The logs and describe routes read only the pods of the named deployment. They answer 404 with the error code kserve_runtime_pod_not_found and param pod_name when the pod does not exist, for example after a redeployment replaced it, and for any other pod in the namespace, including a pod of another model. List the model's pods again to find the current one. Any other failure to read the pod answers 500.
Cluster operators can also inspect the Kubernetes resource directly:
kubectl get llminferenceservice -n kserve
kubectl describe llminferenceservice <model-name> -n kserve
kubectl get inferenceservice -n kserveFor a customer organization's model, use its namespace, tenant-<slug>, in place of kserve.
Related pages
Model as a Service
Deploy models on the platform, add external ones, and serve both through the BSQAI API.
Model access and publishing API
List the models an organization sees, set who may call them, publish a tenant's models to other tenants, and manage their Owner and Grant Managers through the BSQAI API.