# Move models from LiteLLM to model routing (/docs/deployment/playbooks/upgrade-litellm-to-model-routing)



BullSequana AI 1.4.0 serves models through LiteLLM. BullSequana AI 2.0.0 does not deploy LiteLLM. The BSQAI API sends each model request to the model's server itself, through a [model route](/docs/ai/model-as-a-service/cloud-models).

The upgrade rebuilds routes only for the models deployed on the platform. It does not carry LiteLLM's other models over. Until you add such a model's route, requests for the model fail with 404 `model_not_found`, and chat, retrieval, and Agents that use it fail too. A reranker is the exception: retrieval goes on without reranking.

## What changes [#what-changes]

* The models that the Model Installer deployed keep running on KServe. The upgrade gives each of them a route under its deployment's name, once. Its mode, features and limits come from the Global organization's preset for the same model, or stay unset when no preset matches; correct them afterwards.
* Predictive models are called through the BSQAI API's serving endpoints. Their pods accept traffic only from the internal inference gateway, so applications that called a predictive model on the KServe gateway or its predictor service stop reaching it until it is registered again and they switch to its serving endpoint.
* Every other LiteLLM model has to be added again as a route.
* Nobody can call a model until an allow rule in their organization lets them. Every organization that exists at upgrade time gets an All Members allow rule on each rebuilt route. A route that you add has none. See [Let members use the models](#let-members-use-the-models).
* LiteLLM's URL, admin UI, and keys stop working. Applications call models through the BSQAI API, with BSQAI API keys.
* Embedding and rerank responses are no longer cached for 24 hours. Every request reaches the model server.
* Speech-to-text and text-to-speech do not change. They use [Speaches](/docs/ai/components/speaches), not LiteLLM.

## Before the upgrade [#before-the-upgrade]

### Record LiteLLM's models [#record-litellms-models]

Record what LiteLLM serves while it still runs. LiteLLM answers at `https://litellm.<base domain>`, or under the prefix in `LITELLM_URL_PREFIX` when the site sets it. Its admin key is `LITELLM_PROXY_MASTER_KEY` in `generated_secrets.env`. In the checkout that holds the site inputs, run:

```bash
export LITELLM="https://litellm.<base domain>"
export LITELLM_ADMIN_KEY="$(sed -n "s/^LITELLM_PROXY_MASTER_KEY=//p" generated_secrets.env | tr -d "'\"")"

curl -sS "$LITELLM/v1/model/info" \
  -H "Authorization: Bearer $LITELLM_ADMIN_KEY" > litellm-models.json

jq -r '.data[] | [.model_name, .litellm_params.model, (.litellm_params.api_base // ""), (.model_info.mode // "")] | @tsv' litellm-models.json
```

The second command prints each model's name, its LiteLLM model, its API base, and its mode. `model_info` also holds the limits and capabilities that LiteLLM listed. The file holds no API keys, so have each provider's key at hand.

For each model that the Model Installer deployed, also keep the settings that you deployed it with: source, runtime profile, mode, features, limits, and vLLM arguments.

### Record embedding input limits [#record-embedding-input-limits]

In 1.4.0, a tenant's settings can hold its embedding model's input limit as `embedding.max_input_tokens`. In 2.0.0, the limit is the **Max input tokens** of the embedding model's route, and the tenant setting is gone: the settings API no longer returns it. For each tenant, as a user with `tenant.read` and `tenant.embedding.read`:

```bash
export BSQAI_API="https://<bsqai-api-host>"
export TOKEN="<access-token>"

curl -sS "$BSQAI_API/v1/config" \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Tenant-Id: <tenant-id>" \
  | jq '{embedding: .embedding.model, max_input_tokens: .embedding.max_input_tokens}'
```

The recorded values show which embedding models need a limit. The limit belongs to the model, not the tenant: after the upgrade, set the **Max input tokens** of each of those models' routes to the model's own input limit, from its model card. If the model's limit is unknown, use the smallest value any tenant recorded for it. Set it in the registration request for a model that the Model Installer deployed, or on the route for any other model. Without it, texts are split at 2,048 tokens.

### Finish Deep Research runs [#finish-deep-research-runs]

A Deep Research run that is still in progress during the upgrade can fail after it. Let running research finish, or cancel it, before you upgrade.

## After the upgrade [#after-the-upgrade]

Model routes are managed in **Studio**, under **Model routing**, in the Global organization, with `platform.global_service.manage`.

### Register installed models again [#register-installed-models-again]

Register again each LLM and predictive deployment that still has no model route, with the same name and the settings that you deployed it with. **Available models** marks such a deployment **Deployed**, and its detail page says **No model route**. Registration adds the model's route. It also applies the deployment again, so KServe restarts the model's pods if anything in the deployment changed.

For a predictive model, send the same registered model name and version and the same serving runtime. Registration adds its route and serving endpoint, and runs 2 replicas unless you set `minReplicas`. Then point its applications at the serving endpoint, as [Use predictive models via API](/docs/develop/use-predictive-models-via-api) describes.

* In the Portal, open **Deploy a model**, then **Advanced setup**, and enter the same deployment name, namespace, model source, runtime, resource profile, model task, limits, and arguments. Compare the configuration preview with `kubectl get llminferenceservice <name> -n kserve -o yaml`.
* With the API, send the same request again to `POST /v1/model-installer/register_model`. See [Model Installer API](/docs/ai/model-as-a-service/model-installer-api).

Do not add a route by hand under an installed model's name. Registration then fails with 409 `model_route_alias_taken`, and the installer removes the model's deployment.

### Add the other models again [#add-the-other-models-again]

Add a route for each other model in `litellm-models.json`, with the same model name. See [Add a model in Studio](/docs/ai/model-as-a-service/cloud-models#add-a-model-in-studio).

First add one connection for each provider resource or server. The prefix of `litellm_params.model` shows the provider. Choose the protocol **Anthropic-compatible** for `anthropic/`, and **OpenAI-compatible** for all the others:

| LiteLLM model                  | Base URL                                                    | API                                  |
| ------------------------------ | ----------------------------------------------------------- | ------------------------------------ |
| `azure/<deployment>`           | `api_base` without its final `/`, followed by `/openai/v1`  | **Chat Completions**                 |
| `azure/responses/<deployment>` | The same as for `azure/`                                    | **Responses**                        |
| `openai/<model>`               | `api_base`, or `https://api.openai.com/v1` if there is none | **Chat Completions**                 |
| `hosted_vllm/<model>`          | `api_base`, which ends in `/v1`                             | **Chat Completions**                 |
| `openrouter/<model>`           | `https://openrouter.ai/api/v1`                              | **Chat Completions**                 |
| `anthropic/<model>`            | `api_base`, or `https://api.anthropic.com` if there is none | **Messages**, with the mode **Chat** |

Then add each model's route:

| LiteLLM                                                                             | Model route                                                                                                    |
| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `model_name`                                                                        | **Model name**                                                                                                 |
| `litellm_params.model` after its prefix                                             | **Upstream model**. For `openrouter/~deepseek/deepseek-flash-latest`, it is `~deepseek/deepseek-flash-latest`. |
| `model_info.mode`                                                                   | **Mode**                                                                                                       |
| `model_info.max_tokens`, `max_input_tokens`, and `max_output_tokens`                | The limits of the same names                                                                                   |
| `model_info.supports_vision`, `supports_function_calling`, and `supports_reasoning` | **Image input**, **Tool calling**, and **Reasoning**. Where LiteLLM has no value, leave **Not stated**.        |
| `litellm_params.timeout` and `max_retries`                                          | The connection's **Timeout in seconds** and **Max retries**                                                    |
| `litellm_params.tags`                                                               | **Tags**                                                                                                       |

LiteLLM's other `model_info` fields have no route equivalent. LiteLLM also filled in some capabilities from its own list of known models, so make sure that each value is true for your deployment.

A route states only the capabilities that you enter. Set image input to **Yes** for each model that reads images, or chat refuses image attachments for it.

Keep each model's name. Tenant settings, Agents, and API clients refer to models by name, and each tenant's settings record its embedding model by name. With the same name, retrieval keeps using the existing collection, without a reindex.

### Let members use the models [#let-members-use-the-models]

Give each organization's members access to the models they need, under **Access** on each model's row in **Available models**. For everyone in an organization, set **All Members** to **Can use** and select **Save**. Include the embedding model and the reranker that the organization's settings name: retrieval calls them as the member who asks. See [Model access](/docs/use/portal/ai-hub#model-access).

* In the Global organization, a platform administrator sets the rules.
* In each customer organization, its Tenant Administrator sets them.

Until then, the organization's members get 404 `model_not_found` for the model, and chat, retrieval, and Agents that use it fail. Set the rules before the checks below, and run the checks as a user who may use the models.

### Check the result [#check-the-result]

1. **Available models** lists every model that you added, and marks only speech models, if any, as **Deployed**. Any other deployment marked **Deployed** has no route named after it, or you may not call that route. A model listed as **Not enabled for you** has no rule that lets you use it.

2. Check that each model a tenant's settings name is in **Available models**. For each tenant, as a user with `tenant.read` and `tenant.embedding.read`:

   ```bash
   export BSQAI_API="https://<bsqai-api-host>"
   export TOKEN="<access-token>"

   curl -sS "$BSQAI_API/v1/config" \
     -H "Authorization: Bearer $TOKEN" \
     -H "X-Tenant-Id: <tenant-id>" \
     | jq '{default: .chat.default_selection, title: .chat.auto_rename_model, embedding: .embedding.model, reranker: .reranker.model, summarizer: .compaction.summarizer_model, image_description: (.vision.image_description_model // .document_processing.image_description_model)}'
   ```

   Without `tenant.embedding.read`, the response hides the embedding settings, and `embedding` shows `null` even when a model is set.

3. Send a chat message with each chat model, and ask a question about a file with retrieval.

4. Retrieval goes on without reranking when the reranker fails, so check the reranker by itself. For each tenant whose settings name a reranker:

   ```bash
   curl -sS "$BSQAI_API/v1/rerank" \
     -H "Authorization: Bearer $TOKEN" \
     -H "X-Tenant-Id: <tenant-id>" \
     -H "Content-Type: application/json" \
     -d '{
       "query": "What is the capital of France?",
       "documents": ["Berlin is the capital of Germany.", "Paris is the capital of France."]
     }'
   ```

   The response ranks the two documents. 404 `model_not_found` means that the reranker has no route.

Files that were uploaded while the embedding model had no route can fail to index. [Reprocess](/docs/ai/rag#reprocess) them.

### Move API clients [#move-api-clients]

Applications that called LiteLLM directly must call the BSQAI API instead, with a BSQAI API key and the same model names. See [Use Local Models via API](/docs/develop/use-local-models-via-api).

## Related pages [#related-pages]

* [Upgrade Temporal to 1.32](/docs/deployment/playbooks/upgrade-temporal-1-32)
* [Models in Studio](/docs/use/portal/ai-hub)
