Use predictive models via API
Call classic machine learning and deep learning models deployed on BullSequana AI through the BSQAI API.
This page is for application developers who send predictions to a deployed predictive model, such as a scikit-learn, XGBoost, or MLflow model. The model must already be deployed. A data scientist or platform administrator deploys it from the registry, as Deploy a predictive model describes.
Applications call predictive models through the BSQAI API, never on the model servers behind it. The BSQAI API authenticates each call, sends it to the model's server, and turns the server's errors into BSQAI API errors. The model servers accept traffic only from the platform's internal gateway.
What you need
- The BSQAI API endpoint of your deployment, for example
https://api.<platform-domain>/v1. - A JWT or a BullSequana API key, as Use local models via API describes. Calls need the
tenant.readpermission. - An allow rule that lets you use the model in your organization. A Tenant Administrator, or the model's owner or one of its Grant Managers, sets it with Access on the model in Studio → Available models; see Model access. An API key uses the models its owner may use.
- The model's serving endpoint ID.
export BSQAI_BASE_URL="https://api.<platform-domain>/v1"
export BSQAI_TOKEN="sk-bsq-..."Find the serving endpoint
Each deployed predictive model has a serving endpoint, identified by an ID. In the Portal, its page under Studio → Available models shows it as Endpoint ID, with example requests and a form to try one; see Predictive models. GET /v1/models/info lists it as model_id, with mode set to predictive, when you may use it:
curl -s "$BSQAI_BASE_URL/models/info" \
-H "Authorization: Bearer $BSQAI_TOKEN" \
| jq '.[] | select(.mode == "predictive") | {model_name, model_id}'export ENDPOINT_ID="<model_id>"The ID stays the same when the model is deployed again with a new version under the same name, so applications can keep it in their configuration.
Serving endpoint operations
| Method and path | Use |
|---|---|
GET /v1/serving-endpoints/{id} | The model's name, platform, inputs, and outputs. For an MLflow model, these are its signature. |
GET /v1/serving-endpoints/{id}/ready | Whether the model is loaded and ready. |
POST /v1/serving-endpoints/{id}/infer | A prediction in the KServe V2 (Open Inference) protocol. |
POST /v1/serving-endpoints/{id}/invocations | A prediction in MLflow's scoring format. |
GET, POST, or PUT /v1/serving-endpoints/{id}/http/{path} | A request to a custom serving container's own HTTP API. |
A model served by MLServer or Triton offers the first four operations. A model on a custom container whose runtime declares its own HTTP API offers only the last one. Any other combination answers 400 with model_api_unsupported. The chat, Responses, embeddings, and rerank APIs also refuse predictive models with model_api_unsupported.
Inspect the model
Read the model's inputs before you build a request:
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID" \
-H "Authorization: Bearer $BSQAI_TOKEN"{
"name": "churn-model",
"inputs": [
{"name": "tenure", "datatype": "FP64", "shape": [-1]},
{"name": "monthly_charges", "datatype": "FP64", "shape": [-1]}
],
"outputs": [{"name": "output-1", "datatype": "INT64", "shape": [-1, 1]}]
}GET /v1/serving-endpoints/{id}/ready answers 200 when the model can take requests.
Send a prediction
KServe V2 protocol
Send one named tensor per input. This request scores two rows:
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/infer" \
-H "Authorization: Bearer $BSQAI_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"inputs": [
{"name": "tenure", "shape": [2], "datatype": "FP64", "data": [3, 48]},
{"name": "monthly_charges", "shape": [2], "datatype": "FP64", "data": [89.5, 20.1]}
]
}'{"model_name": "churn-model", "id": "...", "outputs": [{"name": "output-1", "shape": [2, 1], "datatype": "INT64", "data": [1, 0]}]}For large tensors, such as images, use the V2 binary tensor extension. Send the JSON header followed by the raw tensor bytes, with Content-Type: application/octet-stream and the header's length in Inference-Header-Content-Length. A binary answer carries the same header.
MLflow scoring format
An MLflow model also accepts MLflow's scoring format: dataframe_split, dataframe_records, instances, or inputs.
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/invocations" \
-H "Authorization: Bearer $BSQAI_TOKEN" \
-H "Content-Type: application/json" \
-d '{"dataframe_split": {"columns": ["tenure", "monthly_charges"], "data": [[3, 89.5], [48, 20.1]]}}'{"predictions": [1, 0]}Python
import os
import httpx
base_url = os.environ["BSQAI_BASE_URL"]
endpoint_id = os.environ["ENDPOINT_ID"]
headers = {"Authorization": f"Bearer {os.environ['BSQAI_TOKEN']}"}
response = httpx.post(
f"{base_url}/serving-endpoints/{endpoint_id}/invocations",
headers=headers,
json={"dataframe_split": {"columns": ["tenure", "monthly_charges"], "data": [[3, 89.5]]}},
timeout=60,
)
response.raise_for_status()
print(response.json()["predictions"])Custom HTTP API
A custom serving container keeps its own API. The BSQAI API sends the request body, the query string, and the Content-Type and Accept headers to the container's path unchanged:
curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/http/classify?top_k=3" \
-H "Authorization: Bearer $BSQAI_TOKEN" \
-H "Content-Type: image/jpeg" \
--data-binary @photo.jpgThe path cannot contain empty, ., or .. segments.
Limits
- A request body, and the model's answer, can be up to 16 MiB each. Send larger batches as several requests, or score them in a batch job that reads from object storage.
- A request waits for the model as long as the model's connection allows, 600 seconds by default for installed models.
- A model on Triton accepts at most its deployment's batch limit in one request, or 4 when none was set and its batch dimension is variable. A larger batch is refused with
400inference_request_rejected.
Errors and retries
The BSQAI API answers model server failures with the platform error envelope, {"error": {"type", "code", "detail"}}:
| Situation | Status and code |
|---|---|
| The model rejects the input, for example a missing column, a wrong datatype or shape, or invalid JSON | 400 inference_request_rejected. For MLflow's and Triton's input-validation errors, detail carries the model's reason, for example inference request batch-size must be <= 16 for 'image-classifier'. Other reasons are replaced by a fixed message. |
| The request body is over 16 MiB | 413 inference_payload_too_large |
| The endpoint ID is unknown, or no allow rule lets you use the model | 404 serving_endpoint_not_found |
| Your monthly BCU cap, or the cap of the group that funds your API key, is reached | 429 usage_cap_reached, with x-should-retry: false. Do not retry until a tenant administrator raises the cap or the month ends. |
| The model is not loaded, or its server cannot be reached | 503 model_server_unavailable |
| The model fails while answering | 500 model_inference_failed, without the server's details |
| The model's answer is over 16 MiB | 500 model_answer_too_large |
The BSQAI API retries a request with growing pauses when the model server cannot be reached or reports that the model is not loaded yet. It retries a GET request after 429, 502, 503, or 504 as well. It never sends a request with a body twice once the model may have received it, so a prediction does not run twice. /ready is never retried: it answers at once.
A model that is undeployed stops answering within 10 seconds, with 404 serving_endpoint_not_found.
Usage and caps
/infer, /invocations and /http/... calls are checked against your monthly BCU cap, and against the cap of your API key's funding group, before they reach the model. They are recorded as calls in your usage. Predictive models report no tokens, so their calls add no token BCU. Reading the model with GET /v1/serving-endpoints/{id} or /ready is neither checked nor recorded. See Manage monthly BCU allowances and caps.