Kafka

Component Category

Event streaming

Component Description

Kafka is the event streaming component used in Data for carrying and distributing streams of data between producers and consumers. In BullSequana AI, Kafka is deployed and managed through the Strimzi operator.

Why It Is Used

In BullSequana AI, Kafka provides the backbone for event-driven data movement and decoupled producer-consumer architectures.

Learn More

• Apache Kafka documentation
• Strimzi documentation
• apache/kafka on GitHub
• strimzi/strimzi-kafka-operator on GitHub

Deployment notes

BullSequana AI 1.3.0 deploys Kafka 4.1.1 through Strimzi 0.51.0 in KRaft mode. One KafkaNodePool runs three combined controller and broker nodes with a 100Gi persistent volume per node. Platform defaults set the replication factor to three and the minimum in-sync replica count to two.

Kafka exposes three internal listeners:

• plaintext on port 9092
• TLS on port 9093
• TLS with Keycloak OAuth and the OAUTHBEARER SASL mechanism on port 9094

The Strimzi Entity Operator includes both the Topic Operator and User Operator. Administrators create KafkaTopic and KafkaUser resources when an application needs them; the base platform does not require pre-created topics or users.

Interacts With

• Keycloak, which validates users and services on the OAuth listener.
• Milvus, which uses the internal TLS listener as its message queue.
• Tuples, which can publish authorization audit events when Kafka auditing is enabled.
• Platform applications, which exchange events through operator-managed Kafka topics.