Observability and audit

Tenant-aware telemetry, authorized queries, BCU consumption, and operational evidence.

Agentic Friendly

Foundation collects infrastructure and application metrics, logs, and traces. Operators use Grafana to investigate health, latency, failures, and consumption; tenant telemetry access is authorized separately from infrastructure administration.

Signal paths

SignalCollection and routingStorage and use
MetricsPrometheus scraping and application OTLP through Alloy; Prometheus remote write to Mimir, routed per seriesPrometheus and Grafana; tenant-routed Mimir
Infrastructure pod logsKubernetes log collection through per-node Alloy, under platform scopeLoki
Tenant-aware application logsAuthenticated OTLP with server-authorized tenant contextTenant-routed Loki
Application tracesAuthenticated OTLP, routed by each span's tenant contextTenant-routed Tempo

Trusted tenant routing

Two gateways carry tenant telemetry. The Alloy OTLP gateway, alloy-gateway, receives application logs, traces, and metrics and routes them by tenant. The telemetry query gateway, telemetry-query-envoy, authorizes every Grafana query against Loki, Tempo, and Mimir.

The Alloy OTLP gateway accepts mTLS-authenticated service exporters. Application logs carry the server-authorized tenant UUID in the log-record attribute tenant_id; trace spans carry it in tenant.id. A shared process can serve several tenants, so trace routing is per span rather than based on the pod's resource attributes.

Metrics follow the same rule per series. Allowlisted tenant-serving metrics carry the tenant UUID in the tenant_id label. The Mimir tenant router stores a series without tenant_id, or with tenant_id="platform", in the platform scope, and a series with a lowercase tenant UUID in that tenant's scope. A series with any other tenant_id value goes to quarantine. See Grafana Mimir for the per-tenant limits and alerts.

Client-supplied baggage is not an authorization source. Missing or malformed OTLP tenant context goes to quarantine, not the platform tenant. Quarantine is excluded from ordinary tenant and global query scopes. Routing validates the identifier's shape; the trusted producer is responsible for attaching the authorized identity.

Per-node collection sends infrastructure stdout/stderr to the platform scope. It does not infer authorization from an arbitrary tenant_id written into a log body. Pods annotated with telemetry.bsqai.io/logs-exporter: otlp are excluded from that collection to avoid ingesting their tenant-aware logs twice.

Export and query prerequisites

Platform operators configure the OTLP gateway's server certificate, client CA, and approved exporters' client certificates. Keep the service's CA trust and endpoint hostname aligned; ordinary unauthenticated OTLP exporters cannot use this path.

Grafana log, trace, and Mimir metric queries pass through the telemetry query gateway. It validates a Keycloak token for the telemetry-query audience and resolves the permitted storage scope through the BSQAI API authorizer. Caller-supplied X-Scope-OrgID is not a tenant-access grant. Tenant queries require the organization.telemetry.read or platform.audit.read admission permission, and the requested Organization must be one of the caller's Keycloak Organization memberships in the token. Neither permission opens the tenant view of an Organization the caller does not belong to. Global and platform views require platform authority.

This does not make all Prometheus metrics tenant-private. Mimir stores tenant-routed copies of metric series, while local Prometheus keeps every series. Infrastructure and capacity metrics have their own dashboard and data-source access controls. Test both an allowed tenant query and a denied foreign-tenant query when changing telemetry configuration.

Tenant Grafana Organizations

When the telemetry query gateway serves tenant routes, the tenant operator gives each tenant its own Grafana Organization. Its Loki and Tempo data sources, and a metrics data source where one is configured, are pinned to that tenant. The operator does not add members, so tenant members cannot sign in to it in 2.0.0. They read their usage and BCU spend in the Portal instead, under My usage and, with usage permission, Administration > Analytics. The platform's cross-tenant data sources stay in Grafana's main Organization, which Global Organization members use. See Tenant Grafana Organizations.

Browser telemetry

Portal browser telemetry is disabled by default. Set PORTAL_TELEMETRY_ENABLED=true only with the authenticated Portal export path, BSQAI API OTLP proxy, certificates, and tenant authorization configured. Browser exports travel through that authenticated path; browsers do not receive service-exporter private keys or choose a trusted storage tenant themselves.

After enablement, make a Portal request in a known tenant and verify its authorized log/trace path. Inspect quarantine counters when telemetry is missing instead of widening tenant access to find it.

Read BCU consumption

The BCU Spend Grafana dashboard reports Bull Compute Units from metered node capacity-hours and in-cluster vLLM token counters. It provides month-to-date consumption, rate and cumulative views, tenant allocation, and reconciliation. Month boundaries use UTC.

  1. Open BCU Spend and choose the reporting range.
  2. Review Metered nodes and excluded Management nodes before interpreting totals. Capacity metering is not the same as instantaneous CPU or GPU utilization.
  3. Compare GPU, CPU/memory, and token contributions. Token counters cover input and generated tokens from in-cluster vLLM; external-provider token traffic is outside this meter.
  4. Inspect Attribution by tenant and the Reconciliation panel. Tenant resource attribution uses requests of running pods on metered nodes, not a direct measurement of each tenant's hardware utilization.
Attribution rowMeaning
Tenant OrganizationCapacity allocated through the tenant's namespaces: a workspace profile labeled with the organization, or a namespace the tenant operator created for the tenant
globalThe platform's model-serving namespaces, whose models the Global Organization runs
platformThe platform's other namespaces
unassignedA workspace profile with no tenant association
unallocatedMetered capacity not requested by workloads
unattributedToken consumption not assigned to a tenant; the current token accounting uses this row

The operator configures CPU, memory, GPU-class, and token conversion rates in the Grafana component's [bcu] configuration. The default token conversion is one BCU per million tokens. The default GPU-class rates are set relative to the H100 reference of 10 BCU per GPU-hour, and include A100 (80 GB) at 6, A100 (40 GB) at 5, and T4 at 2; a metered GPU that matches no configured class is billed at the reference rate. Bundle and overage panels use configured values; they do not establish a customer's commercial terms. Confirm the site's rate configuration and measurement coverage before exporting a consumption report.

A substantial reconciliation difference, missing labels, failed recording rules, or a scrape gap needs investigation before using the totals. Save the time range and rate configuration with an exported report.

Token BCU in the usage summaries

The BSQAI API's usage summaries also report token BCU, measured differently from the dashboard's token row:

BCU Spend token rowUsage summaries tokens.bcu
SourceIn-cluster vLLM token countersThe usage record of each model call
CoversTokens served in the clusterEvery routed model call, external providers included
Attributed tounattributedThe consuming tenant and user
Month boundariesUTCEurope/London

Both use the token rate of the Grafana component's [bcu] configuration. The two figures overlap for in-cluster models, so never add them together.

Capacity BCU in the usage records

The BSQAI API also stores capacity BCU, from the same metering rules as the dashboard's capacity rows:

BCU Spend capacity rowsBSQAI API capacity BCU
SourceLive Prometheus recording rulesClosed intervals of those rules, stored per tenant by the usage capacity collector
Missing dataA scrape gap or missing labels changes the totalsAn interval that could not be read stays unrecorded: unknown, never zero. It is read again for up to seven days.
Later changesTotals follow metric retention and the current ratesA stored interval does not change with metric retention or a later rate change
Month boundariesUTCEurope/London

Tenant allowances, user and group caps, the Tenant BCU statement in Administration → Analytics, and My usage use the BSQAI API figures, not the dashboard. See Usage budgets for how each balance is counted.

Operational telemetry and audit

Metrics show health and capacity, logs explain component behavior, and traces connect service calls. MLflow also retains its own model and prompt traces.

The BSQAI API writes one audit log line per agent sandbox action, with the tenant, chat, acting user, and outcome, into the tenant-routed logs.

Operational telemetry is separate from the Data authorization service's durable, per-tenant Kafka audit log. Use that log when a subject needs direct, revocable access to authorization-event history.

On this page