Kafka consumer groups CLI tutorial

Create a consumer group in Kafka from the CLI: run multiple kafka-console-consumer instances sharing one group and watch partition load distribute across them.

Learn how Kafka consumer groups work through hands-on CLI practice in 15 minutes

Consumer groups enable parallel message processing by distributing partitions among group members. This tutorial demonstrates consumer group behavior using the kafka-console-consumer CLI.

What you'll learn:

• How to create multiple consumers in the same group
• How partitions are distributed among consumers
• What happens when consumers join or leave a group
• How offsets are committed and resumed

Use CLI commands with appropriate extensions for your platform: for Windows - kafka-console-consumer.bat, for Mac and Linux - kafka-console-consumer.sh.

In addition, we will provide an optional consumer group parameter with --group flag.

How to create consumers in a Kafka consumer group?

To start consumers in a consumer group, do the following:

1. Create a topic with at least 2 partitions and send data to it
2. Create a first kafka-console-consumer and assign a group name with --group
3. Open a new terminal / shell window
4. Create a second kafka-console-consumer and use the same --group argument
5. Send data to the topic and you will see consumers sharing the reads

If you need a refresh on how consumers in a consumer group work, have a read here.

Create consumer group example

You cannot have more consumers in a group than partitions in your Kafka topic, and therefore we first need to create a Kafka topic with a few partitions (in the example 3).

kafka-topics --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1

Then launch a consumer in a consumer group, named my-first-application

kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application

Open a new terminal/shell window and launch a second consumer in the same consumer group my-first-application (note we're using the exact same command)

kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application

Open a new terminal/shell window and launch a third consumer in the same consumer group my-first-application

kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application

Each consumer in the consumer group my-first-application will get assigned a partition. Produce a few string messages in the topic.

$ kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic
>first message
>second message
>third message
>fourth message

Each consumer will show only the messages produced on the partition that are assigned to it.

If you stop a consumer, messages automatically get sent to the remaining consumers because consumers in a consumer group automatically perform a consumer rebalance.

Stop all consumers

And keep on producing to the topic

>eigth message
>ninth message
>tenth message

Upon restart of a consumer in the group, the consumer will read from the latest committed offsets and read only the messages you've just produced

$ kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application

eigth message
ninth message
tenth message

You have seen how consumers work in consumer groups!

Gotchas

• If you consume in a consumer groups using the --group command, then if you try using the --from-beginning option afterwards with the same group, it will be ignored. Instead, you need to reset your consumer groups as shown here.
• If you don't specify a --group option, the consumer group of the consumer will be a random consumer group such as console-consumer-11984
• If you see one consumer getting all the messages, that probably means that your topic was only created with 1 partition, which you can verify with the kafka-topics --describe command

Consumer group behavior summary

Scenario	Result
3 consumers, 3 partitions	Each consumer gets 1 partition
2 consumers, 3 partitions	One consumer gets 2 partitions
4 consumers, 3 partitions	One consumer sits idle
Consumer leaves group	Partitions rebalanced to remaining consumers
Consumer joins group	Rebalance distributes partitions

See it in practice with Conduktor

Conduktor Console displays consumer group membership, partition assignments, and lag metrics in real-time. Monitor rebalances and track consumer health visually.

Next steps

• Manage and reset consumer group offsets with kafka-consumer-groups
• Understand incremental rebalancing for production
• Configure consumer settings for stability