Reference architecture
A high-level view of how AI services fit together.
AI combines product-facing interfaces with tenant-aware application services, model serving, retrieval, tools, and observability.
Architecture view
The backend file-event worker owns document ingestion: Docling converts documents, while the worker caches artifacts, requests embeddings and writes Milvus vectors. Live voice uses Speaches and the selected model, not the retrieval or tool paths. Deep Research runs as its own Temporal worker that calls the tenant's web-search provider and the research model.
The BSQAI API routes each model request through the model's route, to in-cluster inference or to a configured external model server. Before it forwards the first call of a request, it checks that the caller may use the model and that the monthly BCU caps of the member, and of an API key's funding group, are not reached. It then records each call as usage. A refused call to a model the caller may not use answers like a call to an unknown model.
Predictive models are deployed differently but called the same way: the API deploys a registered MLflow version with a compatible python_function flavor to an MLServer runtime, or one with the onnx flavor to a Triton runtime, through a KServe InferenceService and adds its route. Applications call it through the BSQAI API by its serving endpoint, in the KServe V2 protocol or MLflow's scoring format, and the call reaches the model through the private inference gateway. See Use predictive models via API.
The Global organization's models run in the kserve namespace. A customer organization's models run in its tenant namespace, tenant-<slug>. KAI Scheduler places their pods in the tenant's GPU queue, within the guarantee and cap that a platform administrator sets, and the namespace admits only the serving runtimes and resource profiles in the tenant's deployment policy.
Key patterns
- The AI Web Portal is the main user surface; the BSQAI API is the application integration boundary.
- Keycloak Organizations select the tenant, and OpenFGA enforces resource access inside that tenant.
- Nothing is usable by default: a model, including a Global one, needs an allow rule for the caller, apart from its owner. Model access does not depend on the client: the Portal, API keys, and service accounts follow the same rules.
- Usage records carry the tenant, user, API key, application, model, and group of each call, without prompts or response content. Analytics, My usage, and Budgets read them.
- Temporal runs durable tenant, file, reindex, Deep Research, skill publication, and managed-ingestion workflows.
- The BSQAI API stores a Managed MCP Server as desired state. The MCP reconcile worker writes the
MCPServerresource, MCP Lifecycle Operator runs the workload, and the MCP Gateway carries authenticated calls to managed or remote servers. - The agent sandbox is optional. Each sandbox is a Kata Containers micro-VM in the
coreai-sandboxesnamespace; the BSQAI API starts one per chat and saves its workspace in the organization's object storage. - Application OTLP signals flow through Grafana Alloy into Foundation observability services; KServe contributes scraped metrics rather than generic OTLP traces or logs.
AI domains
| Domain | Main role | Example components |
|---|---|---|
| User experience | Provide chat, Build, files, Agents, Skills, Connectors, Deep Research, voice, Studio, and administration | AI Web Portal |
| API and orchestration | Expose tenant-aware application services and workflows | BSQAI API, Temporal |
| Tenant lifecycle | Reconcile tenant identity, namespace, quota, GPU queues, network policies, and optional database, storage, private MLflow, and Grafana organization | BSQAI Tenant Operator, PlatformTenant |
| Model lifecycle | Track experiments, register, share, and deploy models, in the installation's MLflow or an organization's private MLflow | Model Installer, MLflow |
| Model access and inference | Route and execute model requests | BSQAI API model routing, KServe, vLLM, MLServer, Triton, KAI Scheduler |
| Model access governance | Publish models, grant their use, and delegate Grant Managers | BSQAI API serving-endpoint access, OpenFGA |
| Usage and budgets | Record each model call with its BCU cost, report usage, and enforce monthly caps | BSQAI API usage records, Analytics, My usage, Budgets |
| Tools and search | Connect approved MCP servers and web-search providers | MCP Gateway, MCP Lifecycle Operator, BSQAI API web-search providers |
| Code execution | Run agent and skill scripts in isolated sandboxes, where the site enables them | Agent Sandbox controller, Kata Containers |
| Retrieval and documents | Convert, embed, store, and retrieve content | Backend file-event worker, Docling, Milvus |
| Voice conversations | Transcribe speech, generate replies and synthesize audio | BSQAI API, Speaches, selected model |
| AI observability | Trace application and inference operations | OpenTelemetry, MLflow, Grafana Alloy |
Design intent
Applications integrate with the Portal or the BSQAI API for platform operations, LLM access, and predictive model serving endpoints. They call LLMs through /v1/chat/completions, /v1/responses, or the Anthropic Messages API under /anthropic, and predictive models through /v1/serving-endpoints/{id}. Each call is checked against the model's allow rules and the monthly BCU caps, and recorded as usage. See Usage budgets and Usage analytics. Internal model, storage, workflow, and tool components can evolve without changing those primary product interfaces. Tenant identity and authorization remain attached as requests move through each layer.