Chapters

Kafka Consumer Group Management CLI Tutorial

Learn to work with consumer groups in Kafka using CLI


The Kafka Consumer Groups CLI kafka-consumer-groups is used to manage consumer groups in Kafka. Make sure you have started Kafka beforehand.

CLI Extensions

Use CLI commands with appropriate extensions for your platform, e.g., kafka-consumer-groups.bat for windows, kafka-consumer-groups.sh for Linux


How to reset a Kafka consumer group using the CLI?

To reset a Kafka consumer groups, we need to:

  • Find your broker hostname and port e.g., localhost:9092

  • Understand the offset reset strategy (to earliest, to latest, to specific offset, shift by...)

  • Stop the running consumer groups (otherwise the command will fail)

  • Use the kafka-consumer-groups.sh CLI with the --reset-offsets option

Example: Reset offsets to the earliest

First, ensure that the consumers are stopped ("has no active members")

1 2 3 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-application Consumer group 'my-first-application' has no active members.

Observe the current offsets for your consumer group (same command as above)

1 2 3 4 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-first-application first_topic 0 3 3 0 - - - my-first-application first_topic 1 5 5 0 - - - my-first-application first_topic 2 6 6 0 - - -

We will reset the offsets to the earliest position in order to read the topic entirely again

1 2 3 4 5 6 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-application --reset-offsets --to-earliest --execute --topic first_topic GROUP TOPIC PARTITION NEW-OFFSET my-first-application first_topic 0 0 my-first-application first_topic 1 0 my-first-application first_topic 2 0

As you can see the new offsets for that consumer group for all partitions are 0, which means that upon restarting a consumer in that group, it will read from the beginning of each partition:

1 2 3 4 5 6 7 8 9 10 11 12 13 kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --group my-first-application third message fifth message seventh message tenth message first message fourth message eigth message hello world second message sixth message ninth message

Note that messages are read in order for each partition, not across partitions (we have 3 partitions in this example) - we will never stop reminding you this.

Example: Reset offsets shift by

Stop the running consumers to be able to reset offsets.

Shift by allows you to rewind offsets by a specific value (negative to go back in messages and positive to advance in messages).

1 2 3 4 5 6 7 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-application Consumer group 'my-first-application' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-first-application first_topic 0 3 3 0 - - - my-first-application first_topic 1 5 5 0 - - - my-first-application first_topic 2 6 6 0 - - -

In this example we reset offsets by shifting by -2 for the consumer group my-first-application subscribed on the topic first_topic

1 2 3 4 5 6 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-first-application --reset-offsets --shift-by -2 --execute --topic first_topic GROUP TOPIC PARTITION NEW-OFFSET my-first-application first_topic 0 1 my-first-application first_topic 1 3 my-first-application first_topic 2 4

As you can see the offsets decreased by 2 for each partition.

Read messages from the topic first_topic using Kafka console consumer CLI. It will only return last 2 messages from each partition of the topic.

1 2 3 4 5 6 7 kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --group my-first-application seventh message tenth message fourth message eigth message sixth message ninth message

Gotchas

Here are the common mistakes and caveats with the kafka-consumer-groups.sh command:

  • You cannot reset a consumer group if consumers are active in it.

  • This command can be used to reprocess data for a consumer group (in case you have a bug fix)

  • This command be also be used to advance message consumption in Kafka (for example if a message is a poison pill, or if your consumer is too slow to catch up with the entire topic).

Extra: Important options you can set (advanced)

--all-groups

Applies to all groups, use with caution

--all-topics

Consider all topics assigned to a group in the reset-offsets process, use with caution

--by-duration

Reset to offsets by duration

--dry-run

Only show the expected result, but does not actually run the command

--to-datetime, --by-period, --to-earliest, --to-latest, --shift-by, --from-file, --to-current

All the various options available to you to reset the offsets


How to list all Kafka consumers in a consumer groups using the CLI?

Listing all the Kafka consumer in a consumer group help you understand where the consumers are placed on your network, and how far they are into the topic consumption.

1 2 3 4 5 6 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-first-application GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-first-application first_topic 0 3 3 0 consumer-my-first-application-1-0237b0a1-911d-45f1-891f-8fd7630a7593 /172.19.0.1 consumer-my-first-application-1 my-first-application first_topic 1 5 5 0 consumer-my-first-application-1-70ddc756-8dbc-45e4-b5b6-1e5a75db9e62 /172.19.0.1 consumer-my-first-application-1 my-first-application first_topic 2 6 6 0 consumer-my-first-application-1-a8ce2af3-97b3-4445-bba3-f4a5f6c4464d /172.19.0.1 consumer-my-first-application-1

The CONSUMER ID represents the unique identifier of the consumer to the Kafka broker

The CLIENT ID represents a client-side setting that you can optionally set to identify a consumer in your consumer groups (with the client.id consumer property)

The CURRENT-OFFSET is the latest committed offset for that group

The LOG-END-OFFSET represents the latest message offset available in the topic-partition for consumption

The LAG is the difference of LOG-END-OFFSET and CURRENT-OFFSET and represents how far behind a consumer is to the tail of a topic.

The HOST is the hostname / IP of the consumer client machine.


How to List of all Kafka consumers groups using the CLI?

Listing the consumer groups help you understand which ones could be down, or not stable.

Listing consumer groups state

1 2 3 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list --state GROUP STATE my-first-application Stable

Describe all consumer groups and state (helpful for assignment strategy and coordinator ID)

1 2 3 4 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups --state GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS my-first-application 127.0.0.1:9092 (1) range Stable 3

You can instead of using --all-groups just specify the command for one group only.

Note:

As of Kafka >= 2.1, the "console-consumer-" groups will not appear in the output


How to delete a consumer group in Kafka using the CLI?

You may want to delete a consumer group in order to reset entirely the reading mechanism. For this, you can use the delete option:

1 2 $ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-first-application Deletion of requested consumer groups ('my-first-application') was successful.

Alternatively, if you want to only delete offsets for a specific topic (helpful when your consumer group is reading from multiple topics) you can use the following command:

1 2 3 4 5 6 7 kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete-offsets --group my-first-application --topic first_topic Request succeed for deleting offsets with topic first_topic group my-first-application TOPIC PARTITION STATUS first_topic 0 Successful first_topic 1 Successful first_topic 2 Successful

Was this content helpful?
4
0
PreviousKafka Consumers in Group CLI Tutorial
NextKafka Connect CLI Tutorial

Conduktor & Kafka Consumer Groups

Now that you have seen how to perform all these tasks using the Kafka Consumer Group CLI, you may wonder how you can achieve this using a graphical UI tool.

Using Conduktor you can manage your consumer groups in Kafka, view the membership of your consumers, and reset your consumer groups to a desired offset

Kafka Consumer Group Screenshot from Conduktor DevTools
Kafka Consumer Groups in Conduktor
Conduktor screenshot showing Kafka Consumer Group details
Kafka Consumer Groups Details
Kafka Consumer Group Reset Process in Conduktor
Kafka Consumer Groups Reset