Kafka Commands Cheat sheet for beginners

Topics

1) Creating a New Topic
kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic my-topic

2) Verify the topic
kafka-topics --list --zookeeper localhost:2181

3) Adding Partitions
kafka-topics --zookeeper localhost:2181 --alter --topic my-topic --partitions 16

4) Deleting a Topic
kafka-topics --zookeeper localhost:2181 --delete --topic my-topic

5) Listing All Topics in a Cluster
kafka-topics --zookeeper localhost:2181 --list

6) Describing Topic Details
kafka-topics --zookeeper localhost:2181/kafka-cluster --describe

7) Show Under-replicated Partitions for topics
kafka-topics --zookeeper localhost:2181/kafka-cluster --describe --under-replicated-partitions

8)  Change topic retention i.e set SLA
kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=28800000*

9) Purge a Topic
kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000
kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --delete-config retention.ms

10) Get the earliest offset still in a topic
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -2

Get the latest offset still in a topic
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1

Producers

1) Produce messages standard input
kafka-console-producer --broker-list localhost:9092 --topic my-topic

2) Produce messages file
kafka-console-producer --broker-list localhost:9092 --topic test < messages.txt

3) Produce Avro messages
kafka-avro-console-producer --broker-list localhost:9092 --topic my.Topic --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}' --property schema.registry.url=http://localhost:8081

And enter a few values from the console:
{"f1": "value1"}

Consumers
Consume messages
1) Start a consumer from the beginning of the log
kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic --from-beginning

2) Consume 1 message
kafka-console-consumer --bootstrap-server localhost:9092 --topic my-topic  --max-messages 1

3) Consume 1 message from __consumer_offsets
kafka-console-consumer --bootstrap-server localhost:9092 --topic __consumer_offsets --formatter 'kafka.coordinator.GroupMetadataManager$OffsetsMessageFormatter' --max-messages 1

4) Consume, specify consumer group
kafka-console-consumer --topic my-topic --new-consumer --bootstrap-server localhost:9092 --consumer-property group.id=my-group

5) Consume Avro messages
kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081 --max-messages 10

kafka-avro-console-consumer --topic position-reports --new-consumer --bootstrap-server localhost:9092 --from-beginning --property schema.registry.url=localhost:8081

Consumers admin operations
1) List Groups
kafka-consumer-groups --new-consumer --list --bootstrap-server localhost:9092

2) Describe Groups
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group testgroup

Config
1) Set the retention for the topic
kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic --add-config retention.ms=3600000

2) Show all configuration overrides for a topic
kafka-configs --zookeeper localhost:2181 --describe --entity-type topics --entity-name my-topic

3) Delete a configuration override for retention.ms for a topic
kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name my-topic --delete-config retention.ms

Performance
Producer
kafka-producer-perf-test --topic position-reports --throughput 10000 --record-size 300 --num-records 20000 --producer-props bootstrap.servers="localhost:9092"

ACLs
kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --consumer --topic topicA --group groupA

kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic topicA

List the ACLs
kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --list --topic topicA

Zookeeper
Enter zookeepr shell:
zookeeper-shell localhost:2182 ls /

kafkacat
Getting the last five message of a topic
kafkacat -C -b localhost:9092 -t mytopic -p 0 -o -5 -e