Kafka

The event streaming component in Data, deployed through Strimzi.

Agentic Friendly

Component Category

Event streaming

Component Description

Kafka is the event streaming component used in Data for carrying and distributing streams of data between producers and consumers. In BullSequana AI, Kafka is deployed and managed through the Strimzi operator.

Why It Is Used

In BullSequana AI, Kafka provides the backbone for event-driven data movement and decoupled producer-consumer architectures.

Learn More

Deployment notes

BullSequana AI deploys Kafka 4.1.2 through Strimzi 1.0.1 in KRaft mode. One KafkaNodePool runs three combined controller and broker nodes with a 100Gi persistent volume per node. Platform defaults set the replication factor to three and the minimum in-sync replica count to two.

Kafka exposes four internal listeners. When network policies are enabled, which is the default, and the cluster enforces them, Strimzi admits only the listed clients to each listener:

PortListenerAuthenticationAdmitted clients
9092plainNonePods labelled bsqai.io/kafka-listener: plain; no platform workload uses it
9093tlsTLS with scram-sha-512Milvus, which uses its milvus KafkaUser, scoped to the by-dev topic prefix, for its write-ahead log
9094oauthTLS with Keycloak OAuth and the OAUTHBEARER SASL mechanismTuples, which publishes tenant audit events
9095auditTLS with Strimzi's native scram-sha-512Tuples pods; used for per-subject audit-consumer credentials

A new producer or consumer needs its own KafkaUser with ACLs and an operator-added listener peer or NetworkPolicy before it can connect.

Authorization is cluster-wide (authorization: simple), so an ACL scopes each identity to only the topics and operations it needs. The Strimzi Entity Operator includes both the Topic Operator and User Operator. The platform creates the topics and users of its own clients: the milvus user, and the audit topic and producer user of each tenant that has Kafka auditing configured. Administrators create other KafkaTopic and KafkaUser resources when an application needs them. Tuples' audit feature also creates a KafkaUser on the fly whenever a subject mints an audit credential, scoped by ACL to read and describe only that subject's own tenant topic, and deletes it again on revocation or through a periodic orphan sweep.

Interacts With

  • Keycloak, which validates users and services on the OAuth listener.
  • Milvus, which uses the internal TLS listener with SCRAM credentials as its write-ahead log.
  • Tuples, which publishes each tenant's authorization audit events to a dedicated topic and mints per-subject, revocable KafkaUser credentials on the SCRAM listener for direct audit consumption. See Authorization Events Audit.
  • Customer applications, which can use Kafka once an operator creates their KafkaTopic, KafkaUser and network access.

On this page