Kafka Topics CLI Tutorial
How to create, delete, describe, or change a Kafka topic?
Kafka Topics CLI, i.e., kafka-topics
is used to create, delete, describe, or change a topic in Kafka. Make sure you have started Kafka beforehand.
Use CLI commands with appropriate extensions for your platform, e.g., kafka-topics.bat
for windows, kafka-topics.sh
for Linux and Mac
To create a Kafka topic, we need to provide the mandatory parameters:
If Kafka v2.2+, use the Kafka hostname and port e.g.,
localhost:9092
If older version of Kafka, use the Zookeeper URL and port e.g.
localhost:2181
Provide the mandatory parameters: topic name, number of partitions and replication factor.
Use the
kafka-topics.sh
CLI with the--create
option
If you have a recent version of Kafka, we strongly recommend you to use the command with the --bootstrap-server
option because the Zookeeper option is now deprecated and is removed as part of Kafka v3.
Creating the Kafka topic first_topic
with 3 partitions and a replication factor of 1 when my Kafka broker is running at localhost:9092
Kafka v2.2+:
1
kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1
Kafka v2.1 or less:
1
kafka-topics.sh --zookeeper localhost:2181 --topic first_topic --create --partitions 3 --replication-factor 1
1
2
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic first_topic.
The WARNING is saying that you shouldn't mix underscores and periods in topic name
Here are the common mistakes and caveats with the kafka-topics.sh --create
command:
You cannot specify a replication factor greater than the number of brokers you have
You can specify as many partitions as you want, 3 is a good number to get started with in dev
If you see the documentation after running a command, this means your command is wrong. Scroll up to see details about the error message
There are no default for partitions and replication factor and you must specify those explicitly
The topic name must contain only ASCII alphanumerics, '.', '_' and '-'
--config
You can set topic-level configurations for example --config max.message.bytes=64000
--disable-rack-aware
Disable rack aware replica assignment (not recommended, set only if you know what you're doing)
To list Kafka topics, we need to provide the mandatory parameters:
If Kafka v2.2+, use the Kafka hostname and port e.g.,
localhost:9092
If older version of Kafka, use the Zookeeper URL and port e.g.
localhost:2181
Use the
kafka-topics.sh
CLI with the--list
option
If you have a recent version of Kafka, we strongly recommend you to use the command with the --bootstrap-server
option because the Zookeeper option is now deprecated and is removed as part of Kafka v3.
Listing topics when my Kafka broker is running at localhost:9092
Kafka v2.2+:
Kafka v2.1 or less:
Here are the common mistakes and caveats with the kafka-topics.sh --list
command:
This command lists internal topics that you won't have created (such as
__consumer_offsets
). Do not try to delete these topics. Use--exclude-internal
if you want to hide these topicsIf you see the documentation after running a command, this means your command is wrong. Scroll up to see details about the error message
--exclude-internal
Exclude internal topics when running list or describe command (they're listed by default)
To filter topics based on replication status:
--at-min-isr-partitions
If set when describing topics, only show partitions whose isr count is equal to the configured minimum.
--unavailable-partitions
Only show partitions whose leader is not available.
--under-min-isr-partitions
Only show partitions whose isr count is less than the configured minimum.
--under-replicated-partitions
Only show under replicated partitions
To describe a Kafka topic and get partition details, we need to provide the mandatory parameters
If Kafka v2.2+, use the Kafka hostname and port e.g.,
localhost:9092
If older version of Kafka, use the Zookeeper URL and port e.g.
localhost:2181
Use the
kafka-topics.sh
CLI with the--describe
option
If you have a recent version of Kafka, we strongly recommend you to use the command with the --bootstrap-server
option because the Zookeeper option is now deprecated and is removed as part of Kafka v3.
Listing topics when my Kafka broker is running at localhost:9092
Kafka v2.2+:
Kafka v2.1 or less:
You can specify a comma delimited list of topics to describe more than one at a time.
1
2
3
4
Topic: first_topic TopicId: fN9mf1UDSmiCdKIDFyMEIQ PartitionCount: 3 ReplicationFactor: 1 Configs: cleanup.policy=delete
Topic: first_topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: first_topic Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: first_topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1
Leader: 1
means that for partition 0, the broker with the ID 1
is the leader.
Replicas: 1
means that for partition 0, the broker with the ID 1
is a replica.
Isr: 1
means that for partition 0, the broker with the ID 1
is an in-sync replica.
In case the command is running against a Kafka cluster, you may see
1
2
3
4
5
Topic: third_topic TopicId: 84yqCErzTG27J4wv44dkPQ PartitionCount: 4 ReplicationFactor: 3 Configs: cleanup.policy=delete
Topic: third_topic Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: third_topic Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: third_topic Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: third_topic Partition: 3 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Leader: 2
means that for partition 0
, the broker with the ID 2
is the leader.
Replicas: 2,3,1
means that for partition 0
, the brokers with the ID 2, 3 and 1 are replicas.
Isr: 2,3,1
means that for partition 0
, the brokers with the ID 2, 3 and 1 are replicas in-sync replica.
Here are the common mistakes and caveats with the kafka-topics.sh --describe
command:
It's easy to get partition ID and broker ID confused. Look at the Command Output section above to make sense of what the numbers mean in the command output
If you see the documentation after running a command, this means your command is wrong. Scroll up to see details about the error message
You can specify a comma delimited list to describe a list of topics
If you don't specify a
--topic
option, it will describe all topics
To describe all topics (usually combined with the options describe after)
Do not set the --topic
argument and you will describe all Kafka topics
To filter partitions based on replication status:
--at-min-isr-partitions
If set when describing topics, only show partitions whose ISR count is equal to the configured minimum.
--unavailable-partitions
Only show partitions whose leader is not available
--under-min-isr-partitions
Only show partitions whose ISR count is less than the configured minimum.
--under-replicated-partitions
Only show under replicated partitions
To increase the number of partitions in a Kafka topic, we need to provide the mandatory parameters.
If Kafka v2.2+, use the Kafka hostname and port e.g.,
localhost:9092
If older version of Kafka, use the Zookeeper URL and port e.g.
localhost:2181
Provide the mandatory parameters: topic name and number of partitions
Use the
kafka-topics.sh
CLI with the--alter
option
Increasing the number of partitions in a Kafka topic a DANGEROUS OPERATION if your applications are relying on key-based ordering. In that case, create a new topic and copy all data there instead to have keys properly re-distributed.
Altering the Kafka topic first_topic
to have 5 partitions when my Kafka broker is running at localhost:9092
Kafka v2.2+:
Kafka v2.1 or less:
The command does not have any output, although you can verify afterwards with a --describe
command.
1
2
3
4
5
6
7
$ kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic first_topic
Topic: first_topic TopicId: fN9mf1UDSmiCdKIDFyMEIQ PartitionCount: 5 ReplicationFactor: 1 Configs: cleanup.policy=delete
Topic: first_topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: first_topic Partition: 1 Leader: 1 Replicas: 1 Isr: 1
Topic: first_topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1
Topic: first_topic Partition: 3 Leader: 1 Replicas: 1 Isr: 1
Topic: first_topic Partition: 4 Leader: 2 Replicas: 2 Isr: 2
Here are the common mistakes and caveats with the kafka-topics.sh --alter
command:
This command is NOT RECOMMENDED to run when your consumers are relying on key-based ordering as changing the number of partitions changes the key-hashing technique
You can only add partitions, not remove partitions
To delete a Kafka topic, we need to provide the mandatory parameters.
If Kafka v2.2+, use the Kafka hostname and port e.g.,
localhost:9092
If older version of Kafka, use the Zookeeper URL and port e.g.
localhost:2181
Use the
kafka-topics.sh
CLI with the--delete
optionEnsure that the Kafka brokers allow for topic deletion
delete.topic.enable=true
(default)
If you have a recent version of Kafka, we strongly recommend you to use the command with the --bootstrap-server
option because the Zookeeper option is now deprecated and is removed as part of Kafka v3.
Deleting the topic first_topic
when my Kafka broker is running at localhost:9092
Kafka v2.2+:
Kafka v2.1 or less:
You can specify a comma delimited list of topics to delete more than one topic at a time
You can verify the outcome of the command with a kafka-topics.sh --describe
command.
It make take some time (depending on the topic size) to see the effect of the command.
Here are the common mistakes and caveats with the kafka-topics.sh --delete
command:
In case topic deletion is not enabled (see the
delete.topic.enable
broker setting) then the topics will be "marked for deletion" but will not be deletedDeleting a topic in Kafka may take some time and this is why the
kafka-topics
command returns an empty output even before the topic is deleted (only the command is sent)You can specify a comma delimited list to delete a list of topics
Windows has a long-standing bug (KAFKA-1194) that makes Kafka crash if you delete topics. The only way to recover from this error is to manually delete the topic folders in data/kafka
.
If you are running Kafka on WSL2, you can safely delete topics
Conduktor & Kafka Topics
Learning the CLI commands for Kafka topics is great practice, but it can be so much easier with a proper tool.
Conduktor Platform enables you to list, create, delete, manage, and configure topics in seconds without any complexity. Try it out for free