Prometheus

Metrics collection and monitoring component for Foundation operations.

Agentic Friendly

Component Category

Monitoring and debugging / metrics

Component Description

Prometheus is an open-source monitoring system and time-series database for collecting and querying metrics.

Why It Is Used

In BullSequana AI Foundation, Prometheus is used to gather operational metrics from infrastructure and platform services, enabling health monitoring, alerting, and performance analysis.

Learn More

Deployment notes

Prometheus deploys as part of the kube-prometheus-stack Helm chart, sharing the grafana namespace at sync wave 9 in the common tier. It runs with 100Gi of persistent storage and 30-day metric retention. AlertManager is disabled by default.

Prometheus also remote-writes every series it ingests to Grafana Mimir through the Mimir tenant router, which stores each series in its tenant's organization. Dashboards and the platform rules keep reading local Prometheus. With Mimir disabled, there is no remote write.

Prometheus discovers ServiceMonitor and PodMonitor resources only in namespaces labelled observability.platform.io/prometheus: "true", which the platform sets on its own namespaces. Tenant and Kubeflow profile namespaces do not carry the label, so their workloads cannot add scrape targets.

Platform rules and readers

Prometheus evaluates two sets of platform recording rules:

RulesPurposeRead by
BCU metering (bcu:*)Configured capacity in BCU per node, GPU class, namespace, and tenant, and in-cluster vLLM token consumption in BCUThe BSQAI API usage capacity job, which samples them every 10 minutes into the usage records behind Administration > Analytics, and the BCU Spend dashboard
GPU attribution (gpu:*)Which pods, tenants, and model deployments use each GPU, device utilization and memory, and per-queue GPU demand, guarantees, and allocationThe Administration > GPU usage page for platform administrators; see Review GPU usage

See Read BCU consumption for how BCU figures are interpreted.

A NetworkPolicy admits only these callers to Prometheus on port 9090:

  • the telemetry query gateway, for Grafana's Prometheus data source;
  • the Alloy gateway and the Tempo metrics generator, which remote-write into Prometheus;
  • the Ceph manager;
  • the BSQAI API and its usage capacity job.

Interacts With

  • Grafana, which consumes Prometheus as the main metrics data source through the telemetry query gateway.
  • BSQAI API, which reads the BCU and GPU rules for Analytics and GPU usage.
  • Grafana Mimir, which receives every series through remote write and stores it per tenant organization.
  • Grafana Alloy and Grafana Tempo, whose gateway and metrics generator remote-write into Prometheus.
  • KAI Scheduler, whose queue metrics feed the GPU attribution rules.
  • Temporal and other platform components, which expose metrics through service monitors.

On this page