Grafana

Visualization and dashboarding component for Foundation observability.

Agentic Friendly

Component Category

Monitoring and debugging / visualization

Component Description

Grafana is the visualization and dashboarding layer for metrics, logs, and traces. BullSequana AI provisions platform dashboards from labeled Kubernetes ConfigMap resources and configures Prometheus, Mimir, Loki, and Tempo as the core data sources.

Why It Is Used

In BullSequana AI Foundation, Grafana gives operators a common interface for platform health, service metrics, logs, and distributed traces. Component-specific dashboards complement the cluster-wide Kubernetes views.

Learn More

Dashboard provisioning

Grafana discovers dashboards from ConfigMap resources labeled with grafana_dashboard=1. Components own their dashboards in their own namespaces, so installing or removing a platform component updates the available dashboard set without changing the Grafana release. Each dashboard's grafana_folder annotation places it in a folder: Kubernetes for the cluster views, and BSQAI/Foundation, BSQAI/Inference, BSQAI/Agent, BSQAI/Data, and BSQAI/Billing for component dashboards. The [dashboards] table of the grafana component names the folders.

Start with the Kubernetes global, namespace, or node views when the affected service is not yet known. Use the component dashboard after narrowing the problem to a specific subsystem, then correlate the metrics with Loki logs or Tempo traces.

Platform dashboard inventory

A dashboard is present only while its component is deployed.

Kubernetes

  • Kubernetes / Views / Global
  • Kubernetes / Views / Namespaces
  • Kubernetes / Views / Nodes
  • Kubernetes / Views / Pods
  • Kubernetes / System / API Server
  • Kubernetes / System / CoreDNS
  • Kubernetes / Persistent Volumes
  • Namespace Monitoring
  • Cilium - Hubble Metrics, which shows data only on clusters that run Cilium with Hubble metrics

BSQAI/Foundation

  • Kubernetes / Storage Usage, with Rook-Ceph
  • CloudNativePG, the single PostgreSQL dashboard for the database clusters and the operator
  • Envoy Gateway
  • cert-manager
  • External Secrets Operator
  • Istio Service Mesh
  • Kyverno Policy Engine
  • Keycloak capacity planning dashboard
  • Keycloak troubleshooting dashboard
  • OpenBao
  • OpenFGA
  • ArgoCD
  • Prometheus
  • Grafana Alloy
  • Loki Kubernetes Logs
  • NVIDIA DCGM Exporter
  • Kubescape telemetry, only while Kubescape is enabled

BSQAI/Inference

  • KServe Controller
  • KServe LLMInferenceService

BSQAI/Agent

  • Milvus - Data Path
  • Milvus - Query Path

BSQAI/Data

  • MLflow Tracking
  • Temporal Server Metrics
  • Strimzi Kafka Operator
  • Argo Events EventBus
  • Argo Workflows Metrics
  • Kubeflow Pipelines

BSQAI/Billing

  • BCU Spend

The BCU Spend dashboard reports configured capacity and in-cluster vLLM token consumption, with tenant attribution and reconciliation. See Read BCU consumption for coverage, rates, and interpretation.

Deployment notes

Grafana 13 deploys as part of the kube-prometheus-stack chart in the grafana namespace at sync wave 9 in the common tier. Grafana uses the CNPG PostgreSQL cluster for its own state and authenticates through Keycloak OIDC.

Access

Grafana sets each user's role at every sign-in from their Keycloak groups:

Keycloak groupGrafana role
platform-administrators, the platform administratorsAdmin
PROAI-METRICS-ADMIN-GROUPAdmin
PROAI-METRICS-USERS-GROUPEditor
Grafana-ViewerViewer

These roles apply in the Platform organization. Admin is that organization's administrator role, not Grafana server administration. A user removed from a group loses its role at their next sign-in.

By default, Grafana refuses a user who is in none of these groups. A deployment can instead map Grafana organizations to Keycloak Organizations, with the [organizations] enforce_org_mapping setting of the grafana component. Then only members of the Global organization reach the Platform organization, so add platform operators to the Global organization before turning it on. There, the groups above set the role, and a Global member in none of them gets Viewer. Grafana refuses a user outside the Global organization, whatever their groups, unless the deployment maps that user's organization to a Grafana organization.

The tenant operator creates a Grafana Organization for each tenant, with its own Loki, Tempo, and Metrics data sources. See Tenant Grafana Organizations. It does not manage membership, so tenant members without one of the groups above cannot sign in to Grafana. Tenant members read their usage and BCU spend in the Portal instead, under My usage and, with usage permission, Administration > Analytics.

The role and the data are separate. Dashboards query their data sources with the signed-in user's token, and the data sources answer only users who hold the platform.audit.read permission. Members of PROAI-METRICS-ADMIN-GROUP hold it. Membership of platform-administrators alone does not grant it.

Metrics endpoint

Grafana exposes its own Prometheus metrics at /metrics and plugin metrics under /metrics/plugins/. Prometheus scrapes /metrics inside the cluster. By default, the public gateway answers both paths with 404, so they are not reachable from outside the cluster.

To read them from outside, for example while debugging Grafana, set GRAFANA_METRICS_EXTERNAL_ACCESS=true (the [metrics] external_access setting of the grafana component), then render and sync. The paths then require HTTP basic authentication with the Grafana admin user and password from the grafana-admin secret. Prometheus scrapes with the same credentials.

Changing the setting restarts Grafana. Grafana scrapes can fail briefly while Prometheus loads the new credentials. Check that the Prometheus query up{job="grafana"} returns 1 again. Set it back to false when you finish debugging.

Interacts With

  • Prometheus, which provides the platform metrics data source.
  • Grafana Mimir, which provides the Mimir data source for the tenant-partitioned metrics store's platform organization.
  • Grafana Loki, which provides the platform log data source.
  • Grafana Tempo, which provides the platform trace data source.
  • Keycloak, which provides OIDC authentication and role mapping.
  • PostgreSQL, which stores Grafana state and configuration.
  • Platform components, which provision their dashboards through labeled ConfigMap resources.

On this page