Reference architecture

A high-level view of how AI services fit together.

Agentic Friendly

AI combines product-facing interfaces with tenant-aware application services, model serving, retrieval, tools, and observability.

Architecture view

The backend file-event worker owns document ingestion: Docling converts documents, while the worker caches artifacts, requests embeddings and writes Milvus vectors. Live voice uses Speaches and the selected model, not the retrieval or tool paths. Deep Research runs as its own Temporal worker that calls the tenant's web-search provider and the research model.

The BSQAI API routes each model request through the model's route, to in-cluster inference or to a configured external model server. Before it forwards the first call of a request, it checks that the caller may use the model and that the monthly BCU caps of the member, and of an API key's funding group, are not reached. It then records each call as usage. A refused call to a model the caller may not use answers like a call to an unknown model.

Predictive models are deployed differently but called the same way: the API deploys a registered MLflow version with a compatible python_function flavor to an MLServer runtime, or one with the onnx flavor to a Triton runtime, through a KServe InferenceService and adds its route. Applications call it through the BSQAI API by its serving endpoint, in the KServe V2 protocol or MLflow's scoring format, and the call reaches the model through the private inference gateway. See Use predictive models via API.

The Global organization's models run in the kserve namespace. A customer organization's models run in its tenant namespace, tenant-<slug>. KAI Scheduler places their pods in the tenant's GPU queue, within the guarantee and cap that a platform administrator sets, and the namespace admits only the serving runtimes and resource profiles in the tenant's deployment policy.

Key patterns

  • The AI Web Portal is the main user surface; the BSQAI API is the application integration boundary.
  • Keycloak Organizations select the tenant, and OpenFGA enforces resource access inside that tenant.
  • Nothing is usable by default: a model, including a Global one, needs an allow rule for the caller, apart from its owner. Model access does not depend on the client: the Portal, API keys, and service accounts follow the same rules.
  • Usage records carry the tenant, user, API key, application, model, and group of each call, without prompts or response content. Analytics, My usage, and Budgets read them.
  • Temporal runs durable tenant, file, reindex, Deep Research, skill publication, and managed-ingestion workflows.
  • The BSQAI API stores a Managed MCP Server as desired state. The MCP reconcile worker writes the MCPServer resource, MCP Lifecycle Operator runs the workload, and the MCP Gateway carries authenticated calls to managed or remote servers.
  • The agent sandbox is optional. Each sandbox is a Kata Containers micro-VM in the coreai-sandboxes namespace; the BSQAI API starts one per chat and saves its workspace in the organization's object storage.
  • Application OTLP signals flow through Grafana Alloy into Foundation observability services; KServe contributes scraped metrics rather than generic OTLP traces or logs.

AI domains

DomainMain roleExample components
User experienceProvide chat, Build, files, Agents, Skills, Connectors, Deep Research, voice, Studio, and administrationAI Web Portal
API and orchestrationExpose tenant-aware application services and workflowsBSQAI API, Temporal
Tenant lifecycleReconcile tenant identity, namespace, quota, GPU queues, network policies, and optional database, storage, private MLflow, and Grafana organizationBSQAI Tenant Operator, PlatformTenant
Model lifecycleTrack experiments, register, share, and deploy models, in the installation's MLflow or an organization's private MLflowModel Installer, MLflow
Model access and inferenceRoute and execute model requestsBSQAI API model routing, KServe, vLLM, MLServer, Triton, KAI Scheduler
Model access governancePublish models, grant their use, and delegate Grant ManagersBSQAI API serving-endpoint access, OpenFGA
Usage and budgetsRecord each model call with its BCU cost, report usage, and enforce monthly capsBSQAI API usage records, Analytics, My usage, Budgets
Tools and searchConnect approved MCP servers and web-search providersMCP Gateway, MCP Lifecycle Operator, BSQAI API web-search providers
Code executionRun agent and skill scripts in isolated sandboxes, where the site enables themAgent Sandbox controller, Kata Containers
Retrieval and documentsConvert, embed, store, and retrieve contentBackend file-event worker, Docling, Milvus
Voice conversationsTranscribe speech, generate replies and synthesize audioBSQAI API, Speaches, selected model
AI observabilityTrace application and inference operationsOpenTelemetry, MLflow, Grafana Alloy

Design intent

Applications integrate with the Portal or the BSQAI API for platform operations, LLM access, and predictive model serving endpoints. They call LLMs through /v1/chat/completions, /v1/responses, or the Anthropic Messages API under /anthropic, and predictive models through /v1/serving-endpoints/{id}. Each call is checked against the model's allow rules and the monthly BCU caps, and recorded as usage. See Usage budgets and Usage analytics. Internal model, storage, workflow, and tool components can evolve without changing those primary product interfaces. Tenant identity and authorization remain attached as requests move through each layer.

On this page