# Use predictive models via API (/docs/develop/use-predictive-models-via-api)



This page is for application developers who send predictions to a deployed predictive model, such as a scikit-learn, XGBoost, or MLflow model. The model must already be deployed. A data scientist or platform administrator deploys it from the registry, as [Deploy a predictive model](/docs/use/portal/ai-hub#deploy-a-predictive-model) describes.

Applications call predictive models through the BSQAI API, never on the model servers behind it. The BSQAI API authenticates each call, sends it to the model's server, and turns the server's errors into BSQAI API errors. The model servers accept traffic only from the platform's internal gateway.

## What you need [#what-you-need]

* The BSQAI API endpoint of your deployment, for example `https://api.<platform-domain>/v1`.
* A JWT or a BullSequana API key, as [Use local models via API](/docs/develop/use-local-models-via-api#authentication-options) describes. Calls need the `tenant.read` permission.
* An allow rule that lets you use the model in your organization. A Tenant Administrator, or the model's owner or one of its Grant Managers, sets it with **Access** on the model in **Studio → Available models**; see [Model access](/docs/use/portal/ai-hub#model-access). An API key uses the models its owner may use.
* The model's serving endpoint ID.

```bash
export BSQAI_BASE_URL="https://api.<platform-domain>/v1"
export BSQAI_TOKEN="sk-bsq-..."
```

## Find the serving endpoint [#find-the-serving-endpoint]

Each deployed predictive model has a serving endpoint, identified by an ID. In the Portal, its page under **Studio → Available models** shows it as **Endpoint ID**, with example requests and a form to try one; see [Predictive models](/docs/use/portal/ai-hub#predictive-models). `GET /v1/models/info` lists it as `model_id`, with `mode` set to `predictive`, when you may use it:

```bash
curl -s "$BSQAI_BASE_URL/models/info" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  | jq '.[] | select(.mode == "predictive") | {model_name, model_id}'
```

```bash
export ENDPOINT_ID="<model_id>"
```

The ID stays the same when the model is deployed again with a new version under the same name, so applications can keep it in their configuration.

## Serving endpoint operations [#serving-endpoint-operations]

| Method and path                                                | Use                                                                                            |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `GET /v1/serving-endpoints/{id}`                               | The model's name, platform, inputs, and outputs. For an MLflow model, these are its signature. |
| `GET /v1/serving-endpoints/{id}/ready`                         | Whether the model is loaded and ready.                                                         |
| `POST /v1/serving-endpoints/{id}/infer`                        | A prediction in the KServe V2 (Open Inference) protocol.                                       |
| `POST /v1/serving-endpoints/{id}/invocations`                  | A prediction in MLflow's scoring format.                                                       |
| `GET`, `POST`, or `PUT /v1/serving-endpoints/{id}/http/{path}` | A request to a custom serving container's own HTTP API.                                        |

A model served by MLServer or Triton offers the first four operations. A model on a custom container whose runtime declares its own HTTP API offers only the last one. Any other combination answers `400` with `model_api_unsupported`. The chat, Responses, embeddings, and rerank APIs also refuse predictive models with `model_api_unsupported`.

## Inspect the model [#inspect-the-model]

Read the model's inputs before you build a request:

```bash
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID" \
  -H "Authorization: Bearer $BSQAI_TOKEN"
```

```json
{
  "name": "churn-model",
  "inputs": [
    {"name": "tenure", "datatype": "FP64", "shape": [-1]},
    {"name": "monthly_charges", "datatype": "FP64", "shape": [-1]}
  ],
  "outputs": [{"name": "output-1", "datatype": "INT64", "shape": [-1, 1]}]
}
```

`GET /v1/serving-endpoints/{id}/ready` answers `200` when the model can take requests.

## Send a prediction [#send-a-prediction]

### KServe V2 protocol [#kserve-v2-protocol]

Send one named tensor per input. This request scores two rows:

```bash
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/infer" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {"name": "tenure", "shape": [2], "datatype": "FP64", "data": [3, 48]},
      {"name": "monthly_charges", "shape": [2], "datatype": "FP64", "data": [89.5, 20.1]}
    ]
  }'
```

```json
{"model_name": "churn-model", "id": "...", "outputs": [{"name": "output-1", "shape": [2, 1], "datatype": "INT64", "data": [1, 0]}]}
```

For large tensors, such as images, use the V2 binary tensor extension. Send the JSON header followed by the raw tensor bytes, with `Content-Type: application/octet-stream` and the header's length in `Inference-Header-Content-Length`. A binary answer carries the same header.

### MLflow scoring format [#mlflow-scoring-format]

An MLflow model also accepts MLflow's scoring format: `dataframe_split`, `dataframe_records`, `instances`, or `inputs`.

```bash
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/invocations" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataframe_split": {"columns": ["tenure", "monthly_charges"], "data": [[3, 89.5], [48, 20.1]]}}'
```

```json
{"predictions": [1, 0]}
```

### Python [#python]

```python
import os

import httpx

base_url = os.environ["BSQAI_BASE_URL"]
endpoint_id = os.environ["ENDPOINT_ID"]
headers = {"Authorization": f"Bearer {os.environ['BSQAI_TOKEN']}"}

response = httpx.post(
    f"{base_url}/serving-endpoints/{endpoint_id}/invocations",
    headers=headers,
    json={"dataframe_split": {"columns": ["tenure", "monthly_charges"], "data": [[3, 89.5]]}},
    timeout=60,
)
response.raise_for_status()
print(response.json()["predictions"])
```

### Custom HTTP API [#custom-http-api]

A custom serving container keeps its own API. The BSQAI API sends the request body, the query string, and the `Content-Type` and `Accept` headers to the container's path unchanged:

```bash
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/http/classify?top_k=3" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "Content-Type: image/jpeg" \
  --data-binary @photo.jpg
```

The path cannot contain empty, `.`, or `..` segments.

## Limits [#limits]

* A request body, and the model's answer, can be up to 16 MiB each. Send larger batches as several requests, or score them in a batch job that reads from object storage.
* A request waits for the model as long as the model's connection allows, 600 seconds by default for installed models.
* A model on Triton accepts at most its deployment's batch limit in one request, or 4 when none was set and its batch dimension is variable. A larger batch is refused with `400` `inference_request_rejected`.

## Errors and retries [#errors-and-retries]

The BSQAI API answers model server failures with the platform error envelope, `{"error": {"type", "code", "detail"}}`:

| Situation                                                                                             | Status and code                                                                                                                                                                                                                                             |
| ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The model rejects the input, for example a missing column, a wrong datatype or shape, or invalid JSON | `400` `inference_request_rejected`. For MLflow's and Triton's input-validation errors, `detail` carries the model's reason, for example `inference request batch-size must be <= 16 for 'image-classifier'`. Other reasons are replaced by a fixed message. |
| The request body is over 16 MiB                                                                       | `413` `inference_payload_too_large`                                                                                                                                                                                                                         |
| The endpoint ID is unknown, or no allow rule lets you use the model                                   | `404` `serving_endpoint_not_found`                                                                                                                                                                                                                          |
| Your monthly BCU cap, or the cap of the group that funds your API key, is reached                     | `429` `usage_cap_reached`, with `x-should-retry: false`. Do not retry until a tenant administrator raises the cap or the month ends.                                                                                                                        |
| The model is not loaded, or its server cannot be reached                                              | `503` `model_server_unavailable`                                                                                                                                                                                                                            |
| The model fails while answering                                                                       | `500` `model_inference_failed`, without the server's details                                                                                                                                                                                                |
| The model's answer is over 16 MiB                                                                     | `500` `model_answer_too_large`                                                                                                                                                                                                                              |

The BSQAI API retries a request with growing pauses when the model server cannot be reached or reports that the model is not loaded yet. It retries a `GET` request after `429`, `502`, `503`, or `504` as well. It never sends a request with a body twice once the model may have received it, so a prediction does not run twice. `/ready` is never retried: it answers at once.

A model that is undeployed stops answering within 10 seconds, with `404` `serving_endpoint_not_found`.

## Usage and caps [#usage-and-caps]

`/infer`, `/invocations` and `/http/...` calls are checked against your monthly BCU cap, and against the cap of your API key's funding group, before they reach the model. They are recorded as calls in your usage. Predictive models report no tokens, so their calls add no token BCU. Reading the model with `GET /v1/serving-endpoints/{id}` or `/ready` is neither checked nor recorded. See [Manage monthly BCU allowances and caps](/docs/administration/usage-budgets).

## Related pages [#related-pages]

* [Use local models via API](/docs/develop/use-local-models-via-api)
* [Deploy a predictive model](/docs/use/portal/ai-hub#deploy-a-predictive-model)
* [Model Installer API](/docs/ai/model-as-a-service/model-installer-api)
* [KServe](/docs/ai/components/kserve)
