Cloud and external models

Serve models that run at a cloud provider or on another server through model routes in the BSQAI API.

Agentic Friendly

A model that runs outside BullSequana AI, at a cloud provider such as Azure OpenAI or Anthropic or on another server, is not deployed with the Model Installer API. A platform administrator adds it as a model route. The BSQAI API then serves the model under the route's model name, to the members that an allow rule lets call it. A route that you add starts with no rule, so nobody can call the model until one exists.

Applications call the model through the BSQAI API by that name, like any other model. The model server's address and API key stay in the BSQAI API.

Model routes and connections

The BSQAI API sends each generation, embedding, and rerank request to the server that the route of the requested model name points to.

RecordWhat it holds
ConnectionA model server: its base URL, its protocol, an API key, how long one request may take, and how many times a failed request is sent again
RouteThe model name that clients send, the connection to the model's server, the upstream model name that server expects, the API the model is called with, the model's mode, and optional limits, capabilities, and tags

Several routes can share one connection, for example the models of one Azure OpenAI resource.

The Model Installer adds a connection and a route for each model that it deploys, and removes both with the deployment. A predictive model's connection speaks KServe's V2 inference protocol or a custom container's HTTP API, and applications call the model by its serving endpoint. Administrators cannot change the installer's records, except to correct what an installed chat model can do and to choose which tenants may use a Global model. Platform administrators add, change, and delete all other connections and routes, which speak the OpenAI or the Anthropic API.

Portal chat, Build, Agents, the Chat Completions (/v1/chat/completions), Responses (/v1/responses), embeddings (/v1/embeddings), rerank (/v1/rerank), and Anthropic-compatible (/anthropic) APIs, retrieval embeddings and reranking, image descriptions, and Deep Research all reach their models through routes. The BSQAI API reads the route on every request, so a change applies to the next request. Speech-to-text and text-to-speech go to Speaches and do not use routes.

Protocols and APIs

A connection speaks one of two protocols. Each route calls its model with an API that its connection's protocol offers.

ProtocolBase URLAPIsModes
OpenAI-compatibleThe URL that paths such as /chat/completions are added to. It usually ends in /v1.Chat Completions or ResponsesAny
Anthropic-compatibleThe server's root URL, which /v1/messages is added toMessagesChat only

Most model servers, vLLM among them, are OpenAI-compatible. For Azure OpenAI and Azure AI Foundry, add /openai/v1 to the resource's endpoint, as in https://<resource>.cognitiveservices.azure.com/openai/v1, and use the deployment name as the upstream model. Choose Responses only for a server that offers OpenAI's Responses API.

Under an OpenAI-compatible base URL, the BSQAI API requests embeddings from /embeddings and reranking from /rerank. A model called with Messages serves chat only. An embedding or rerank request for it is refused with 400 model_api_unsupported.

A connection's protocol cannot change while routes use it.

With the platform's network policies on, the BSQAI API reaches servers outside the cluster only at the addresses in NETWORK_POLICY_EXTERNAL_EGRESS_CIDRS, and only on the ports in LLM_BACKEND_NETWORK_POLICY_EXTERNAL_PORTS, 443 and 80 by default. A model server on another port, such as https://<host>:8000/v1, fails Test connection with "Could not reach the model server" until the platform operator adds the port.

A KServe deployment that serves LoRA adapters has one connection for the base model and all its adapters. Each adapter's route uses that connection and sends the adapter's name as the upstream model, and it takes its API and mode from the base model's route. See Update adapters.

In Portal chat, the BSQAI API ends a turn with the error code chat_content_blocked when the model server reports a block. A block is a message or an answer that the server's safety guardrail rejected, or a refusal by the model. Chats with an Agent and Build chats get the same code. Applications that call the BSQAI API under /v1 or /anthropic do not get this code.

The block codes are content_filter, ResponsibleAIPolicyViolation, invalid_prompt, bio_policy, misalignment_policy_violation, and image_content_policy_violation. On every API, an HTTP 400 error is a block when the error's code, or the innererror.code that Azure OpenAI adds, is a block code. Each API also signals a block in its answer:

APIBlock signal
Chat Completionsfinish_reason is content_filter, or a refusal instead of the answer
ResponsesAn incomplete response with the reason content_filter, a refusal output, or a failed response with a block code
Messagesstop_reason is refusal

The BSQAI API does not treat a guardrail error with another status or code as a block. It ends that turn with chat_internal_error, and Portal chat offers Continue and Try again. A refusal that the model writes as ordinary answer text is not a block either. When a response ends early describes what users can do after a block.

Capabilities

A route can state three capabilities of its model. Each one is Yes, No, or Not stated. Not stated means unknown, which is not the same as No.

CapabilityEffect
Image inputChat accepts image attachments only for a model whose image input is Yes.
Tool callingChat sends no tools to a model whose tool calling is No: the composer hides Tools, and the model answers without web search, Connectors, or Skill tools. Chat refuses a request that needs tools with 400 model_tools_unsupported. The model's Use model tab in Studio states whether the model calls tools, which coding agents need.
ReasoningChat shows a reasoning effort control unless reasoning is No. A model whose reasoning is No accepts only the effort none.

Chat sends tools to a model whose tool calling is Not stated. A vLLM model served without --enable-auto-tool-choice and --tool-call-parser refuses them, and chat fails with model_tools_unsupported. Deploy the model again with those arguments, or set its tool calling to No.

A route can also list the reasoning efforts that the model rejects, from none, minimal, low, medium, high, and xhigh. Chat stops offering them. Chat and the Responses API refuse a request with one of them with 400 reasoning_effort_unsupported, before the request reaches the server. The Anthropic-compatible API does not check its thinking setting against the list. The server decides what every other effort does.

Max tokens, Max input tokens, and Max output tokens are optional. With Max input tokens set, chat compacts a long conversation before it exceeds the model's context window. Studio also writes the input and output limits into the coding-agent configurations on the model's Use model tab.

On an embedding model, Max input tokens also sets how much text the BSQAI API sends to the model at once. The BSQAI API counts each text's tokens with the tokenizer of OpenAI's embedding models. A text longer than three quarters of the limit is split into pieces of that size, and the pieces' vectors are averaged into one vector for the text. The margin covers models whose own tokenizer counts more tokens for the same text. Without Max input tokens, texts are split at 2,048 tokens, so set it for a model that accepts fewer.

Tags

Tags are free text. The tag on-prem lists a model as Local in Studio, and a route without it is listed as External. The Model Installer tags the models it deploys with on-prem by default.

Add a model in Studio

Adding a model requires platform.global_service.manage in the Global organization.

  1. In Studio, open Deploy a model, and under External model select Add external model. The same action on Available models opens the same page.
  2. Under Connection, choose a connection that you added before. For a new server, keep New connection and enter its Name, Protocol (OpenAI-compatible by default), Base URL, and API key. Only administrators see the name. The key is stored encrypted and never shown again; leave it blank for a server that needs no key. Timeout in seconds and Max retries default to 600 and 5.
  3. Select Test connection. The BSQAI API asks the server for its models, which checks the protocol and the base URL. It checks the key only on a server that requires a key to list its models. Some servers, such as OpenRouter, list their models to anyone, and a wrong key fails only at the first model call. When the server lists models, Upstream model offers them.
  4. Under Model, enter the Upstream model, or pick one the server lists, and the Model name that users and clients choose the model by. Model name follows what you type as the upstream model until you change it. Then choose the API and the Mode, which default to Chat Completions and Chat.
  5. Optionally, open Limits, capabilities and tags and fill in what you know about the model.
  6. Select Add model at the top right of the page.
  7. Let members use the model. Select Access on the model's row in Available models, and give All Members, groups, or users Can use, and select Save. Platform administrators set these rules for the Global organization. Each tenant's administrators set them for their own tenant.

Studio External model page with New connection, the OpenAI-compatible protocol, the default timeout and retries, Test connection, and the Model section

The test saves nothing. When it fails, it shows the reason:

MessageCause
"The model server refused the API key. Check the key, or add one if the server needs it."The key is wrong, or the server needs a key and none was entered.
"The server did not answer like an API of the connection's protocol. Check the protocol and the base URL: an OpenAI-compatible one often ends in /v1."The protocol or the base URL is wrong.
"Could not reach the model server, or it failed to answer. Check the base URL, and try again."The server is unreachable from the cluster, did not answer in time, or answered with a server error.

The model then appears in Available models, and the Portal offers it wherever its mode fits, to the members who may call it. A member without a rule cannot call it, and the model is missing from their model pickers. Change or delete it later under Model routing in Studio.

Choose which tenants may use the model

A model that a platform administrator adds may be used by every tenant. To narrow it, select the people button at the end of the route's row under Model routing, then under Who may use it choose All tenants, Chosen tenants, or Owner only. With Chosen tenants, pick at least one tenant under Tenants, then select Save. Each tenant still decides with its own allow rules which of its members may call the model. Model access and publishing API describes the same choice through the API.

Add a model with the API

The model routing endpoints are under /v1/admin. Each one, including the reads, requires platform.global_service.manage.

EndpointUse
GET, POST /v1/admin/model-connectionsList connections, or add one
GET, PUT, DELETE /v1/admin/model-connections/{id}Read, replace, or delete a connection
POST /v1/admin/model-connections/testTest a server before you add it. Nothing is saved.
POST /v1/admin/model-connections/{id}/testTest a saved connection with its stored key
GET, POST /v1/admin/model-routesList routes, or add one
GET, PUT, DELETE /v1/admin/model-routes/{id}Read, replace, or delete a route
PUT /v1/admin/model-routes/{id}/capabilitiesCorrect what an installed chat model can do. Send the route's deployment_revision as you read it.
PUT /v1/admin/model-routes/{id}/audienceChoose which tenants besides its owner may use the model: {"audience": "everyone"}, {"audience": "owner"}, or {"audience": "listed", "tenant_ids": [...]}

Add the connection first. Send a platform administrator's access token as the bearer token:

export BSQAI_API="https://<bsqai-api-host>"
export TOKEN="<platform-administrator-access-token>"

curl -sS "$BSQAI_API/v1/admin/model-connections" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Azure OpenAI Sweden",
    "protocol": "openai",
    "base_url": "https://<resource>.cognitiveservices.azure.com/openai/v1",
    "api_key": "<azure-api-key>"
  }'

The response carries the connection's id. Then add a route to it:

curl -sS "$BSQAI_API/v1/admin/model-routes" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "alias": "gpt-4.1",
    "connection_id": "<connection-id>",
    "upstream_model": "<azure-deployment-name>",
    "api_style": "chat_completions",
    "mode": "chat",
    "supports_vision": true,
    "supports_function_calling": true
  }'
Connection fieldMeaning
nameRequired, up to 255 characters, unique among connections
protocolopenai (the default) or anthropic
base_urlRequired. A user name or password in the URL is refused; send the key as api_key.
api_keyOptional and write-only. Responses carry has_api_key instead. On PUT, omit it to keep the stored key, or send null to remove it.
timeout_seconds1 to 6,000. The default is 600.
max_retries0 to 10. The default is 5.
Route fieldMeaning
aliasRequired. The model name that clients send, up to 255 characters, unique among routes
connection_idRequired. The connection to the model's server
upstream_modelRequired. The name the server expects, such as an Azure deployment name
api_stylechat_completions (the default) or responses on an openai connection, messages on an anthropic one
modechat, completion, embedding, rerank, audio-speech, audio-transcription, image-generation, or realtime. Omit it for an unknown mode. A messages route must be chat.
max_tokens, max_input_tokens, max_output_tokens, vector_dimensionsOptional positive integers
supports_vision, supports_function_calling, supports_reasoningtrue, false, or null for not stated
unsupported_reasoning_effortsThe efforts that the model rejects
tagsFree text. on-prem lists the model as local.

Responses also carry owner_tenant_id, the organization that owns the model, or null for the Global organization, and audience and audience_tenant_ids, which tenants besides the owner may use it. A route that you add has the audience everyone. Change the audience only with the audience endpoint.

A PUT replaces the whole record. The most common errors are:

Status and codeCause
400 model_server_refused_keyA test was refused the key, or the server needs one
400 model_server_unexpected_responseA test got an answer that is not a model list, usually because of the base URL
503 model_server_unavailableA test could not reach the server, or the server failed to answer
409 model_connection_name_taken, model_route_alias_takenAnother connection has the name, or another route has the model name
409 model_route_api_style_unsupportedThe connection's protocol does not offer the route's API
409 model_connection_in_useRoutes still use the connection that you are deleting, or whose protocol you are changing
409 model_managed_by_installerThe record belongs to the Model Installer
409 model_claim_fixed_by_deployment, model_route_claims_not_correctable, model_route_deployment_changedA capability correction changes what the deployment's flags set, is not for an installed chat model, or is for a deployment that changed since you read the route

Usage and budgets

Every model call through a route, to an external model or an installed one, is recorded as usage for the calling tenant, member, and API key, and counts in BCU. A member, or the group that funds an API key, whose monthly cap is reached gets 429 usage_cap_reached, with the OpenAI error type insufficient_quota on the OpenAI-compatible APIs. A tenant over its allowance keeps working. See Manage monthly BCU allowances and caps and Usage analytics.

Embedding and rerank models

Add an embedding or rerank model with an OpenAI-compatible connection and the mode Embedding or Rerank.

There is no vector dimension to configure for retrieval. When a tenant administrator selects an embedding model, the BSQAI API measures its dimension with one embedding. See Document processing and reindexing.

Each tenant's settings record its embedding model by name. Keep an embedding model's name: selecting another name for a tenant starts a reindex.

Naming

  • The model name is the value that clients send as model. No two routes can share one.
  • Keep names stable. Tenant settings, Agents, and API clients refer to models by name. After a rename or a deletion, their requests fail with 404 model_not_found.
  • Choose a name that reads well to users, such as gpt-4.1, rather than a provider's deployment ID. The upstream model holds the name the server expects.

Common mistakes

  • A base URL that ends in /chat/completions or /v1/messages. Enter the URL that those paths are added to.
  • A user name or password in the base URL. Enter the key under API key.
  • Responses for a server that offers only Chat Completions.
  • Not stated image input for a model that reads images. Chat then refuses image attachments for it.
  • An Anthropic-compatible connection for an embedding or rerank model.

On this page