Observability and audit
Tenant-aware telemetry, authorized queries, BCU consumption, and operational evidence.
Foundation collects infrastructure and application metrics, logs, and traces. Operators use Grafana to investigate health, latency, failures, and consumption; tenant telemetry access is authorized separately from infrastructure administration.
Signal paths
| Signal | Collection and routing | Storage and use |
|---|---|---|
| Metrics | Prometheus scraping and application OTLP through Alloy; Prometheus remote write to Mimir, routed per series | Prometheus and Grafana; tenant-routed Mimir |
| Infrastructure pod logs | Kubernetes log collection through per-node Alloy, under platform scope | Loki |
| Tenant-aware application logs | Authenticated OTLP with server-authorized tenant context | Tenant-routed Loki |
| Application traces | Authenticated OTLP, routed by each span's tenant context | Tenant-routed Tempo |
Trusted tenant routing
Two gateways carry tenant telemetry. The Alloy OTLP gateway, alloy-gateway, receives application logs, traces, and metrics and routes them by tenant. The telemetry query gateway, telemetry-query-envoy, authorizes every Grafana query against Loki, Tempo, and Mimir.
The Alloy OTLP gateway accepts mTLS-authenticated service exporters. Application logs carry the server-authorized tenant UUID in the log-record attribute tenant_id; trace spans carry it in tenant.id. A shared process can serve several tenants, so trace routing is per span rather than based on the pod's resource attributes.
Metrics follow the same rule per series. Allowlisted tenant-serving metrics carry the tenant UUID in the tenant_id label. The Mimir tenant router stores a series without tenant_id, or with tenant_id="platform", in the platform scope, and a series with a lowercase tenant UUID in that tenant's scope. A series with any other tenant_id value goes to quarantine. See Grafana Mimir for the per-tenant limits and alerts.
Client-supplied baggage is not an authorization source. Missing or malformed OTLP tenant context goes to quarantine, not the platform tenant. Quarantine is excluded from ordinary tenant and global query scopes. Routing validates the identifier's shape; the trusted producer is responsible for attaching the authorized identity.
Per-node collection sends infrastructure stdout/stderr to the platform scope. It does not infer authorization from an arbitrary tenant_id written into a log body. Pods annotated with telemetry.bsqai.io/logs-exporter: otlp are excluded from that collection to avoid ingesting their tenant-aware logs twice.
Export and query prerequisites
Platform operators configure the OTLP gateway's server certificate, client CA, and approved exporters' client certificates. Keep the service's CA trust and endpoint hostname aligned; ordinary unauthenticated OTLP exporters cannot use this path.
Grafana log, trace, and Mimir metric queries pass through the telemetry query gateway. It validates a Keycloak token for the telemetry-query audience and resolves the permitted storage scope through the BSQAI API authorizer. Caller-supplied X-Scope-OrgID is not a tenant-access grant. Tenant queries require the organization.telemetry.read or platform.audit.read admission permission, and the requested Organization must be one of the caller's Keycloak Organization memberships in the token. Neither permission opens the tenant view of an Organization the caller does not belong to. Global and platform views require platform authority.
This does not make all Prometheus metrics tenant-private. Mimir stores tenant-routed copies of metric series, while local Prometheus keeps every series. Infrastructure and capacity metrics have their own dashboard and data-source access controls. Test both an allowed tenant query and a denied foreign-tenant query when changing telemetry configuration.
Tenant Grafana Organizations
When the telemetry query gateway serves tenant routes, the tenant operator gives each tenant its own Grafana Organization. Its Loki and Tempo data sources, and a metrics data source where one is configured, are pinned to that tenant. The operator does not add members, so tenant members cannot sign in to it in 2.0.0. They read their usage and BCU spend in the Portal instead, under My usage and, with usage permission, Administration > Analytics. The platform's cross-tenant data sources stay in Grafana's main Organization, which Global Organization members use. See Tenant Grafana Organizations.
Browser telemetry
Portal browser telemetry is disabled by default. Set PORTAL_TELEMETRY_ENABLED=true only with the authenticated Portal export path, BSQAI API OTLP proxy, certificates, and tenant authorization configured. Browser exports travel through that authenticated path; browsers do not receive service-exporter private keys or choose a trusted storage tenant themselves.
After enablement, make a Portal request in a known tenant and verify its authorized log/trace path. Inspect quarantine counters when telemetry is missing instead of widening tenant access to find it.
Read BCU consumption
The BCU Spend Grafana dashboard reports Bull Compute Units from metered node capacity-hours and in-cluster vLLM token counters. It provides month-to-date consumption, rate and cumulative views, tenant allocation, and reconciliation. Month boundaries use UTC.
- Open BCU Spend and choose the reporting range.
- Review Metered nodes and excluded Management nodes before interpreting totals. Capacity metering is not the same as instantaneous CPU or GPU utilization.
- Compare GPU, CPU/memory, and token contributions. Token counters cover input and generated tokens from in-cluster vLLM; external-provider token traffic is outside this meter.
- Inspect Attribution by tenant and the Reconciliation panel. Tenant resource attribution uses requests of running pods on metered nodes, not a direct measurement of each tenant's hardware utilization.
| Attribution row | Meaning |
|---|---|
| Tenant Organization | Capacity allocated through the tenant's namespaces: a workspace profile labeled with the organization, or a namespace the tenant operator created for the tenant |
global | The platform's model-serving namespaces, whose models the Global Organization runs |
platform | The platform's other namespaces |
unassigned | A workspace profile with no tenant association |
unallocated | Metered capacity not requested by workloads |
unattributed | Token consumption not assigned to a tenant; the current token accounting uses this row |
The operator configures CPU, memory, GPU-class, and token conversion rates in the Grafana component's [bcu] configuration. The default token conversion is one BCU per million tokens. The default GPU-class rates are set relative to the H100 reference of 10 BCU per GPU-hour, and include A100 (80 GB) at 6, A100 (40 GB) at 5, and T4 at 2; a metered GPU that matches no configured class is billed at the reference rate. Bundle and overage panels use configured values; they do not establish a customer's commercial terms. Confirm the site's rate configuration and measurement coverage before exporting a consumption report.
A substantial reconciliation difference, missing labels, failed recording rules, or a scrape gap needs investigation before using the totals. Save the time range and rate configuration with an exported report.
Token BCU in the usage summaries
The BSQAI API's usage summaries also report token BCU, measured differently from the dashboard's token row:
| BCU Spend token row | Usage summaries tokens.bcu | |
|---|---|---|
| Source | In-cluster vLLM token counters | The usage record of each model call |
| Covers | Tokens served in the cluster | Every routed model call, external providers included |
| Attributed to | unattributed | The consuming tenant and user |
| Month boundaries | UTC | Europe/London |
Both use the token rate of the Grafana component's [bcu] configuration. The two figures overlap for in-cluster models, so never add them together.
Capacity BCU in the usage records
The BSQAI API also stores capacity BCU, from the same metering rules as the dashboard's capacity rows:
| BCU Spend capacity rows | BSQAI API capacity BCU | |
|---|---|---|
| Source | Live Prometheus recording rules | Closed intervals of those rules, stored per tenant by the usage capacity collector |
| Missing data | A scrape gap or missing labels changes the totals | An interval that could not be read stays unrecorded: unknown, never zero. It is read again for up to seven days. |
| Later changes | Totals follow metric retention and the current rates | A stored interval does not change with metric retention or a later rate change |
| Month boundaries | UTC | Europe/London |
Tenant allowances, user and group caps, the Tenant BCU statement in Administration → Analytics, and My usage use the BSQAI API figures, not the dashboard. See Usage budgets for how each balance is counted.
Operational telemetry and audit
Metrics show health and capacity, logs explain component behavior, and traces connect service calls. MLflow also retains its own model and prompt traces.
The BSQAI API writes one audit log line per agent sandbox action, with the tenant, chat, acting user, and outcome, into the tenant-routed logs.
Operational telemetry is separate from the Data authorization service's durable, per-tenant Kafka audit log. Use that log when a subject needs direct, revocable access to authorization-event history.