Kafka Cluster Startup

For the proper formation of the controller quorum, the startup order of the Kafka cluster nodes is crucial. First, the nodes with the controller role must be started to establish the quorum and elect a leader. After that, the nodes with the broker role can be launched.

Start the Kafka service on all nodes, beginning with the controller nodes:

systemctl enable --now kafka

At this stage, the initial Kafka configuration is complete.

Next, for your setup, you need to create topics and configure ACLs.

Below is an example of topic creation using ACL. All actions are performed on the first broker node, which contains the configuration file /app/certs/adminkfk.properties. To enable clients to produce and consume data from topics, you need to create Kafka. users. These usernames will correspond to the CN values in the client certificates, as defined by the mapping rule specified in the ssl.principal.mapping.rules parameter in the broker.properties. file. This setting must be applied on every broker node.

In the example below, the rules map the CN from client certificates to the corresponding Kafka:

ssl.principal.mapping.rules=RULE:^CN=producer_hosts[1-2],O=Work,L=Dubai,ST=Dubai,C=AE$/producer_user/L,RULE:^CN=consumer_hosts[1-3],O=Work,L=Dubai,ST=Dubai,C=AE$/consumer_user/L

Note!

The root certificates of the clients and Kafka must match.

View the list of existing topics:

$ JAVA_HOME=/app/jdk/ /app/kafka/bin/kafka-topics.sh --command-config /app/certs/adminkfk.properties --bootstrap-server 192.168.0.54:9092 –list

Create a topic:

$ JAVA_HOME=/app/jdk/ /app/kafka/bin/kafka-topics.sh  --command-config /app/certs/adminkfk.properties  --bootstrap-server 192.168.0.54:9092 --create --topic test_topic  --partitions 5 --replication-factor 5

Delete a topic:

$ JAVA_HOME=/app/jdk/ /app/kafka/bin/kafka-topics.sh  --command-config /app/certs/adminkfk.properties  --delete --bootstrap-server 192.168.0.54:9092 --topic test_topic 

To grant write permissions for topics test-topic and test-topic1, we will create a user producer-user.

The username is specified with the User parameter, and the topic name is set via --topic. The --allow-host parameter defines the hosts from which the users listed in --allow-principal are allowed access. The --producer attribute indicates that the settings are for clients producing records to the topic. Multiple topics can be specified in a single command, or * can be used for the --topic parameter to grant access to all topics.

Example command:

$ JAVA_HOME=/app/jdk/ /app/kafka/bin/kafka-acls.sh \
--command-config /app/certs/adminkfk.properties \
--bootstrap-server 192.168.0.54:9092 \
--add \
--allow-principal User:producer-user \
--allow-host 192.168.0.71 \
--allow-host 192.168.0.72 \
--producer \
--topic test-topic \
--topic test-topic1

To grant read permissions for topic test-topic, we will create a user consumer-user.

The username is specified with the User parameter, and the topic name is set via --topic. The --allow-host parameter defines the allowed source hosts for the users listed in --allow-principal. The --consumer attribute indicates that the permissions are for consuming data, and the --group parameter specifies the consumer group name used for parallel processing. Multiple topics or groups can be specified in a single command. Using * for --topic grants access to all topics. At least one consumer-group must be specified when configuring ACLs.

Example command:

$ JAVA_HOME=/app/jdk/ /app/kafka/bin/kafka-acls.sh \
--command-config /app/certs/adminkfk.properties \
--bootstrap-server 192.168.0.54:9092 \
--add \
--allow-principal User:consumer-user \
--allow-host 192.168.0.81 \
--allow-host 192.168.0.82 \
--consumer \
--group test-group1 \
--topic test-topic

To check the ACL settings for a specific topic, use the following command:

$ JAVA_HOME=/app/jdk/ /app/kafka/bin/kafka-topics.sh \
--command-config /app/certs/adminkfk.properties \
--bootstrap-server 192.168.0.54:9092 \
--describe \
--topic test-topic