Kafka
The event streaming component in Data, deployed through Strimzi.
Component Category
Event streaming
Component Description
Kafka is the event streaming component used in Data for carrying and distributing streams of data between producers and consumers. In BullSequana AI, Kafka is deployed and managed through the Strimzi operator.
Why It Is Used
In BullSequana AI, Kafka provides the backbone for event-driven data movement and decoupled producer-consumer architectures.
Learn More
- Apache Kafka documentation
- Strimzi documentation
- apache/kafka on GitHub
- strimzi/strimzi-kafka-operator on GitHub
Deployment notes
BullSequana AI deploys Kafka 4.1.2 through Strimzi 1.0.1 in KRaft mode. One KafkaNodePool runs three combined controller and broker nodes with a 100Gi persistent volume per node. Platform defaults set the replication factor to three and the minimum in-sync replica count to two.
Kafka exposes four internal listeners. When network policies are enabled, which is the default, and the cluster enforces them, Strimzi admits only the listed clients to each listener:
| Port | Listener | Authentication | Admitted clients |
|---|---|---|---|
9092 | plain | None | Pods labelled bsqai.io/kafka-listener: plain; no platform workload uses it |
9093 | tls | TLS with scram-sha-512 | Milvus, which uses its milvus KafkaUser, scoped to the by-dev topic prefix, for its write-ahead log |
9094 | oauth | TLS with Keycloak OAuth and the OAUTHBEARER SASL mechanism | Tuples, which publishes tenant audit events |
9095 | audit | TLS with Strimzi's native scram-sha-512 | Tuples pods; used for per-subject audit-consumer credentials |
A new producer or consumer needs its own KafkaUser with ACLs and an operator-added listener peer or NetworkPolicy before it can connect.
Authorization is cluster-wide (authorization: simple), so an ACL scopes each identity to
only the topics and operations it needs. The Strimzi Entity Operator includes both the
Topic Operator and User Operator. The platform creates the topics and users of its own
clients: the milvus user, and the audit topic and producer user of each tenant that has
Kafka auditing configured. Administrators create other KafkaTopic and KafkaUser
resources when an application needs them. Tuples' audit feature also creates a KafkaUser
on the fly whenever a subject mints an audit credential, scoped by ACL to read and
describe only that subject's own tenant topic, and deletes it again on revocation or
through a periodic orphan sweep.
Interacts With
Keycloak, which validates users and services on the OAuth listener.Milvus, which uses the internal TLS listener with SCRAM credentials as its write-ahead log.Tuples, which publishes each tenant's authorization audit events to a dedicated topic and mints per-subject, revocableKafkaUsercredentials on the SCRAM listener for direct audit consumption. See Authorization Events Audit.Customer applications, which can use Kafka once an operator creates theirKafkaTopic,KafkaUserand network access.