Use predictive models via API

Call classic machine learning and deep learning models deployed on BullSequana AI through the BSQAI API.

Agentic Friendly

This page is for application developers who send predictions to a deployed predictive model, such as a scikit-learn, XGBoost, or MLflow model. The model must already be deployed. A data scientist or platform administrator deploys it from the registry, as Deploy a predictive model describes.

Applications call predictive models through the BSQAI API, never on the model servers behind it. The BSQAI API authenticates each call, sends it to the model's server, and turns the server's errors into BSQAI API errors. The model servers accept traffic only from the platform's internal gateway.

What you need

  • The BSQAI API endpoint of your deployment, for example https://api.<platform-domain>/v1.
  • A JWT or a BullSequana API key, as Use local models via API describes. Calls need the tenant.read permission.
  • An allow rule that lets you use the model in your organization. A Tenant Administrator, or the model's owner or one of its Grant Managers, sets it with Access on the model in Studio → Available models; see Model access. An API key uses the models its owner may use.
  • The model's serving endpoint ID.
export BSQAI_BASE_URL="https://api.<platform-domain>/v1"
export BSQAI_TOKEN="sk-bsq-..."

Find the serving endpoint

Each deployed predictive model has a serving endpoint, identified by an ID. In the Portal, its page under Studio → Available models shows it as Endpoint ID, with example requests and a form to try one; see Predictive models. GET /v1/models/info lists it as model_id, with mode set to predictive, when you may use it:

curl -s "$BSQAI_BASE_URL/models/info" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  | jq '.[] | select(.mode == "predictive") | {model_name, model_id}'
export ENDPOINT_ID="<model_id>"

The ID stays the same when the model is deployed again with a new version under the same name, so applications can keep it in their configuration.

Serving endpoint operations

Method and pathUse
GET /v1/serving-endpoints/{id}The model's name, platform, inputs, and outputs. For an MLflow model, these are its signature.
GET /v1/serving-endpoints/{id}/readyWhether the model is loaded and ready.
POST /v1/serving-endpoints/{id}/inferA prediction in the KServe V2 (Open Inference) protocol.
POST /v1/serving-endpoints/{id}/invocationsA prediction in MLflow's scoring format.
GET, POST, or PUT /v1/serving-endpoints/{id}/http/{path}A request to a custom serving container's own HTTP API.

A model served by MLServer or Triton offers the first four operations. A model on a custom container whose runtime declares its own HTTP API offers only the last one. Any other combination answers 400 with model_api_unsupported. The chat, Responses, embeddings, and rerank APIs also refuse predictive models with model_api_unsupported.

Inspect the model

Read the model's inputs before you build a request:

curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID" \
  -H "Authorization: Bearer $BSQAI_TOKEN"
{
  "name": "churn-model",
  "inputs": [
    {"name": "tenure", "datatype": "FP64", "shape": [-1]},
    {"name": "monthly_charges", "datatype": "FP64", "shape": [-1]}
  ],
  "outputs": [{"name": "output-1", "datatype": "INT64", "shape": [-1, 1]}]
}

GET /v1/serving-endpoints/{id}/ready answers 200 when the model can take requests.

Send a prediction

KServe V2 protocol

Send one named tensor per input. This request scores two rows:

curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/infer" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {"name": "tenure", "shape": [2], "datatype": "FP64", "data": [3, 48]},
      {"name": "monthly_charges", "shape": [2], "datatype": "FP64", "data": [89.5, 20.1]}
    ]
  }'
{"model_name": "churn-model", "id": "...", "outputs": [{"name": "output-1", "shape": [2, 1], "datatype": "INT64", "data": [1, 0]}]}

For large tensors, such as images, use the V2 binary tensor extension. Send the JSON header followed by the raw tensor bytes, with Content-Type: application/octet-stream and the header's length in Inference-Header-Content-Length. A binary answer carries the same header.

MLflow scoring format

An MLflow model also accepts MLflow's scoring format: dataframe_split, dataframe_records, instances, or inputs.

curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/invocations" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataframe_split": {"columns": ["tenure", "monthly_charges"], "data": [[3, 89.5], [48, 20.1]]}}'
{"predictions": [1, 0]}

Python

import os

import httpx

base_url = os.environ["BSQAI_BASE_URL"]
endpoint_id = os.environ["ENDPOINT_ID"]
headers = {"Authorization": f"Bearer {os.environ['BSQAI_TOKEN']}"}

response = httpx.post(
    f"{base_url}/serving-endpoints/{endpoint_id}/invocations",
    headers=headers,
    json={"dataframe_split": {"columns": ["tenure", "monthly_charges"], "data": [[3, 89.5]]}},
    timeout=60,
)
response.raise_for_status()
print(response.json()["predictions"])

Custom HTTP API

A custom serving container keeps its own API. The BSQAI API sends the request body, the query string, and the Content-Type and Accept headers to the container's path unchanged:

curl -s "$BSQAI_BASE_URL/serving-endpoints/$ENDPOINT_ID/http/classify?top_k=3" \
  -H "Authorization: Bearer $BSQAI_TOKEN" \
  -H "Content-Type: image/jpeg" \
  --data-binary @photo.jpg

The path cannot contain empty, ., or .. segments.

Limits

  • A request body, and the model's answer, can be up to 16 MiB each. Send larger batches as several requests, or score them in a batch job that reads from object storage.
  • A request waits for the model as long as the model's connection allows, 600 seconds by default for installed models.
  • A model on Triton accepts at most its deployment's batch limit in one request, or 4 when none was set and its batch dimension is variable. A larger batch is refused with 400 inference_request_rejected.

Errors and retries

The BSQAI API answers model server failures with the platform error envelope, {"error": {"type", "code", "detail"}}:

SituationStatus and code
The model rejects the input, for example a missing column, a wrong datatype or shape, or invalid JSON400 inference_request_rejected. For MLflow's and Triton's input-validation errors, detail carries the model's reason, for example inference request batch-size must be <= 16 for 'image-classifier'. Other reasons are replaced by a fixed message.
The request body is over 16 MiB413 inference_payload_too_large
The endpoint ID is unknown, or no allow rule lets you use the model404 serving_endpoint_not_found
Your monthly BCU cap, or the cap of the group that funds your API key, is reached429 usage_cap_reached, with x-should-retry: false. Do not retry until a tenant administrator raises the cap or the month ends.
The model is not loaded, or its server cannot be reached503 model_server_unavailable
The model fails while answering500 model_inference_failed, without the server's details
The model's answer is over 16 MiB500 model_answer_too_large

The BSQAI API retries a request with growing pauses when the model server cannot be reached or reports that the model is not loaded yet. It retries a GET request after 429, 502, 503, or 504 as well. It never sends a request with a body twice once the model may have received it, so a prediction does not run twice. /ready is never retried: it answers at once.

A model that is undeployed stops answering within 10 seconds, with 404 serving_endpoint_not_found.

Usage and caps

/infer, /invocations and /http/... calls are checked against your monthly BCU cap, and against the cap of your API key's funding group, before they reach the model. They are recorded as calls in your usage. Predictive models report no tokens, so their calls add no token BCU. Reading the model with GET /v1/serving-endpoints/{id} or /ready is neither checked nor recorded. See Manage monthly BCU allowances and caps.

On this page