Kafka Consumer Group Management CLI Tutorial
Learn to work with consumer groups in Kafka using CLI
The Kafka Consumer Groups CLI kafka-consumer-groups is used to manage consumer groups in Kafka. Make sure you have started Kafka beforehand.
Use CLI commands with appropriate extensions for your platform, e.g.,
kafka-consumer-groups.bat for windows,
kafka-consumer-groups.sh for Linux
To reset a Kafka consumer groups, we need to:
Find your broker hostname and port e.g.,
Understand the offset reset strategy (to earliest, to latest, to specific offset, shift by...)
Stop the running consumer groups (otherwise the command will fail)
kafka-consumer-groups.shCLI with the
First, ensure that the consumers are stopped ("has no active members")
Observe the current offsets for your consumer group (same command as above)
We will reset the offsets to the earliest position in order to read the topic entirely again
As you can see the new offsets for that consumer group for all partitions are 0, which means that upon restarting a consumer in that group, it will read from the beginning of each partition:
Note that messages are read in order for each partition, not across partitions (we have 3 partitions in this example) - we will never stop reminding you this.
Stop the running consumers to be able to reset offsets.
Shift by allows you to rewind offsets by a specific value (negative to go back in messages and positive to advance in messages).
In this example we reset offsets by shifting by
-2 for the consumer group
my-first-application subscribed on the topic
As you can see the offsets decreased by 2 for each partition.
Read messages from the topic first_topic using Kafka console consumer CLI. It will only return last 2 messages from each partition of the topic.
Here are the common mistakes and caveats with the
You cannot reset a consumer group if consumers are active in it.
This command can be used to reprocess data for a consumer group (in case you have a bug fix)
This command be also be used to advance message consumption in Kafka (for example if a message is a poison pill, or if your consumer is too slow to catch up with the entire topic).
Applies to all groups, use with caution
Consider all topics assigned to a group in the reset-offsets process, use with caution
Reset to offsets by duration
Only show the expected result, but does not actually run the command
All the various options available to you to reset the offsets
Listing all the Kafka consumer in a consumer group help you understand where the consumers are placed on your network, and how far they are into the topic consumption.
CONSUMER ID represents the unique identifier of the consumer to the Kafka broker
CLIENT ID represents a client-side setting that you can optionally set to identify a consumer in your consumer groups (with the
client.id consumer property)
CURRENT-OFFSET is the latest committed offset for that group
LOG-END-OFFSET represents the latest message offset available in the topic-partition for consumption
LAG is the difference of
CURRENT-OFFSET and represents how far behind a consumer is to the tail of a topic.
HOST is the hostname / IP of the consumer client machine.
Listing the consumer groups help you understand which ones could be down, or not stable.
Listing consumer groups state
Describe all consumer groups and state (helpful for assignment strategy and coordinator ID)
You can instead of using
--all-groups just specify the command for one group only.
As of Kafka >= 2.1, the "console-consumer-" groups will not appear in the output
You may want to delete a consumer group in order to reset entirely the reading mechanism. For this, you can use the
Alternatively, if you want to only delete offsets for a specific topic (helpful when your consumer group is reading from multiple topics) you can use the following command:
Conduktor & Kafka Consumer Groups
You've seen how to perform all these tasks using the CLI, but there's no need to keep using such a painful method.
Conduktor Platform provides an easy way to manage your consumer groups in Kafka, view the membership of your groups, and reset your consumer groups to a desired offset. Try it now