Move models from LiteLLM to model routing
Record LiteLLM's models before the BullSequana AI 2.0.0 upgrade, and serve them again through model routes after it.
BullSequana AI 1.4.0 serves models through LiteLLM. BullSequana AI 2.0.0 does not deploy LiteLLM. The BSQAI API sends each model request to the model's server itself, through a model route.
The upgrade rebuilds routes only for the models deployed on the platform. It does not carry LiteLLM's other models over. Until you add such a model's route, requests for the model fail with 404 model_not_found, and chat, retrieval, and Agents that use it fail too. A reranker is the exception: retrieval goes on without reranking.
What changes
- The models that the Model Installer deployed keep running on KServe. The upgrade gives each of them a route under its deployment's name, once. Its mode, features and limits come from the Global organization's preset for the same model, or stay unset when no preset matches; correct them afterwards.
- Predictive models are called through the BSQAI API's serving endpoints. Their pods accept traffic only from the internal inference gateway, so applications that called a predictive model on the KServe gateway or its predictor service stop reaching it until it is registered again and they switch to its serving endpoint.
- Every other LiteLLM model has to be added again as a route.
- Nobody can call a model until an allow rule in their organization lets them. Every organization that exists at upgrade time gets an All Members allow rule on each rebuilt route. A route that you add has none. See Let members use the models.
- LiteLLM's URL, admin UI, and keys stop working. Applications call models through the BSQAI API, with BSQAI API keys.
- Embedding and rerank responses are no longer cached for 24 hours. Every request reaches the model server.
- Speech-to-text and text-to-speech do not change. They use Speaches, not LiteLLM.
Before the upgrade
Record LiteLLM's models
Record what LiteLLM serves while it still runs. LiteLLM answers at https://litellm.<base domain>, or under the prefix in LITELLM_URL_PREFIX when the site sets it. Its admin key is LITELLM_PROXY_MASTER_KEY in generated_secrets.env. In the checkout that holds the site inputs, run:
export LITELLM="https://litellm.<base domain>"
export LITELLM_ADMIN_KEY="$(sed -n "s/^LITELLM_PROXY_MASTER_KEY=//p" generated_secrets.env | tr -d "'\"")"
curl -sS "$LITELLM/v1/model/info" \
-H "Authorization: Bearer $LITELLM_ADMIN_KEY" > litellm-models.json
jq -r '.data[] | [.model_name, .litellm_params.model, (.litellm_params.api_base // ""), (.model_info.mode // "")] | @tsv' litellm-models.jsonThe second command prints each model's name, its LiteLLM model, its API base, and its mode. model_info also holds the limits and capabilities that LiteLLM listed. The file holds no API keys, so have each provider's key at hand.
For each model that the Model Installer deployed, also keep the settings that you deployed it with: source, runtime profile, mode, features, limits, and vLLM arguments.
Record embedding input limits
In 1.4.0, a tenant's settings can hold its embedding model's input limit as embedding.max_input_tokens. In 2.0.0, the limit is the Max input tokens of the embedding model's route, and the tenant setting is gone: the settings API no longer returns it. For each tenant, as a user with tenant.read and tenant.embedding.read:
export BSQAI_API="https://<bsqai-api-host>"
export TOKEN="<access-token>"
curl -sS "$BSQAI_API/v1/config" \
-H "Authorization: Bearer $TOKEN" \
-H "X-Tenant-Id: <tenant-id>" \
| jq '{embedding: .embedding.model, max_input_tokens: .embedding.max_input_tokens}'The recorded values show which embedding models need a limit. The limit belongs to the model, not the tenant: after the upgrade, set the Max input tokens of each of those models' routes to the model's own input limit, from its model card. If the model's limit is unknown, use the smallest value any tenant recorded for it. Set it in the registration request for a model that the Model Installer deployed, or on the route for any other model. Without it, texts are split at 2,048 tokens.
Finish Deep Research runs
A Deep Research run that is still in progress during the upgrade can fail after it. Let running research finish, or cancel it, before you upgrade.
After the upgrade
Model routes are managed in Studio, under Model routing, in the Global organization, with platform.global_service.manage.
Register installed models again
Register again each LLM and predictive deployment that still has no model route, with the same name and the settings that you deployed it with. Available models marks such a deployment Deployed, and its detail page says No model route. Registration adds the model's route. It also applies the deployment again, so KServe restarts the model's pods if anything in the deployment changed.
For a predictive model, send the same registered model name and version and the same serving runtime. Registration adds its route and serving endpoint, and runs 2 replicas unless you set minReplicas. Then point its applications at the serving endpoint, as Use predictive models via API describes.
- In the Portal, open Deploy a model, then Advanced setup, and enter the same deployment name, namespace, model source, runtime, resource profile, model task, limits, and arguments. Compare the configuration preview with
kubectl get llminferenceservice <name> -n kserve -o yaml. - With the API, send the same request again to
POST /v1/model-installer/register_model. See Model Installer API.
Do not add a route by hand under an installed model's name. Registration then fails with 409 model_route_alias_taken, and the installer removes the model's deployment.
Add the other models again
Add a route for each other model in litellm-models.json, with the same model name. See Add a model in Studio.
First add one connection for each provider resource or server. The prefix of litellm_params.model shows the provider. Choose the protocol Anthropic-compatible for anthropic/, and OpenAI-compatible for all the others:
| LiteLLM model | Base URL | API |
|---|---|---|
azure/<deployment> | api_base without its final /, followed by /openai/v1 | Chat Completions |
azure/responses/<deployment> | The same as for azure/ | Responses |
openai/<model> | api_base, or https://api.openai.com/v1 if there is none | Chat Completions |
hosted_vllm/<model> | api_base, which ends in /v1 | Chat Completions |
openrouter/<model> | https://openrouter.ai/api/v1 | Chat Completions |
anthropic/<model> | api_base, or https://api.anthropic.com if there is none | Messages, with the mode Chat |
Then add each model's route:
| LiteLLM | Model route |
|---|---|
model_name | Model name |
litellm_params.model after its prefix | Upstream model. For openrouter/~deepseek/deepseek-flash-latest, it is ~deepseek/deepseek-flash-latest. |
model_info.mode | Mode |
model_info.max_tokens, max_input_tokens, and max_output_tokens | The limits of the same names |
model_info.supports_vision, supports_function_calling, and supports_reasoning | Image input, Tool calling, and Reasoning. Where LiteLLM has no value, leave Not stated. |
litellm_params.timeout and max_retries | The connection's Timeout in seconds and Max retries |
litellm_params.tags | Tags |
LiteLLM's other model_info fields have no route equivalent. LiteLLM also filled in some capabilities from its own list of known models, so make sure that each value is true for your deployment.
A route states only the capabilities that you enter. Set image input to Yes for each model that reads images, or chat refuses image attachments for it.
Keep each model's name. Tenant settings, Agents, and API clients refer to models by name, and each tenant's settings record its embedding model by name. With the same name, retrieval keeps using the existing collection, without a reindex.
Let members use the models
Give each organization's members access to the models they need, under Access on each model's row in Available models. For everyone in an organization, set All Members to Can use and select Save. Include the embedding model and the reranker that the organization's settings name: retrieval calls them as the member who asks. See Model access.
- In the Global organization, a platform administrator sets the rules.
- In each customer organization, its Tenant Administrator sets them.
Until then, the organization's members get 404 model_not_found for the model, and chat, retrieval, and Agents that use it fail. Set the rules before the checks below, and run the checks as a user who may use the models.
Check the result
-
Available models lists every model that you added, and marks only speech models, if any, as Deployed. Any other deployment marked Deployed has no route named after it, or you may not call that route. A model listed as Not enabled for you has no rule that lets you use it.
-
Check that each model a tenant's settings name is in Available models. For each tenant, as a user with
tenant.readandtenant.embedding.read:export BSQAI_API="https://<bsqai-api-host>" export TOKEN="<access-token>" curl -sS "$BSQAI_API/v1/config" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Tenant-Id: <tenant-id>" \ | jq '{default: .chat.default_selection, title: .chat.auto_rename_model, embedding: .embedding.model, reranker: .reranker.model, summarizer: .compaction.summarizer_model, image_description: (.vision.image_description_model // .document_processing.image_description_model)}'Without
tenant.embedding.read, the response hides the embedding settings, andembeddingshowsnulleven when a model is set. -
Send a chat message with each chat model, and ask a question about a file with retrieval.
-
Retrieval goes on without reranking when the reranker fails, so check the reranker by itself. For each tenant whose settings name a reranker:
curl -sS "$BSQAI_API/v1/rerank" \ -H "Authorization: Bearer $TOKEN" \ -H "X-Tenant-Id: <tenant-id>" \ -H "Content-Type: application/json" \ -d '{ "query": "What is the capital of France?", "documents": ["Berlin is the capital of Germany.", "Paris is the capital of France."] }'The response ranks the two documents. 404
model_not_foundmeans that the reranker has no route.
Files that were uploaded while the embedding model had no route can fail to index. Reprocess them.
Move API clients
Applications that called LiteLLM directly must call the BSQAI API instead, with a BSQAI API key and the same model names. See Use Local Models via API.