How to Start Kafka using Docker?
Start Kafka with Docker and run on all systems.
Running Kafka on Docker has the benefit that this method is going to work for all operating systems including Mac, Windows and Linux. It also helps to switch Kafka versions easily by simply updating the underlying docker-compose file.
You must install Docker with the steps mentioned below to work.
Install Docker Desktop on Mac.
Install Docker Desktop on Windows.
Install Docker on Linux (choose your distro on the lefthand side menu).
For Mac & Windows, if you have installed Docker Desktop, then Docker Compose is included as part of those desktop installs.
For Linux, follow the steps here (and do all the steps)
Docker Compose is a tool for defining and running multi-container Docker applications. As we'll start each of the Kafka services, e.g, zookeeper, broker, etc. in a different Docker container, docker-compose is a perfect tool to do that.
To use docker-compose, we have to define the services that make up our Kafka deployment in docker-compose.yml so they can be run together in an isolated environment.
Go to the GitHub project - https://github.com/conduktor/kafka-stack-docker-compose and clone or download the project. It has docker-compose files for many configurations.
1
2
3
4
5
6
7
8
$ git clone https://github.com/conduktor/kafka-stack-docker-compose.git
Cloning into 'kafka-stack-docker-compose'...
remote: Enumerating objects: 551, done.
remote: Counting objects: 100% (219/219), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 551 (delta 145), reused 140 (delta 83), pack-reused 332
Receiving objects: 100% (551/551), 110.29 KiB | 15.76 MiB/s, done.
Resolving deltas: 100% (375/375), done.Next, go to the folder kafka-stack-docker-compose and list the files.
1
2
3
4
5
6
7
8
9
10
11
$ ls -l
total 60
-rw-rw-r-- 1 ec2-user ec2-user 4719 Nov 16 11:37 full-stack.yml
-rw-rw-r-- 1 ec2-user ec2-user 11345 Nov 16 11:37 LICENSE
-rw-rw-r-- 1 ec2-user ec2-user 8100 Nov 16 11:37 README.md
-rwxrwxr-x 1 ec2-user ec2-user 2150 Nov 16 11:37 test.sh
-rw-rw-r-- 1 ec2-user ec2-user 4759 Nov 16 11:37 zk-multiple-kafka-multiple-schema-registry.yml
-rw-rw-r-- 1 ec2-user ec2-user 4209 Nov 16 11:37 zk-multiple-kafka-multiple.yml
-rw-rw-r-- 1 ec2-user ec2-user 2728 Nov 16 11:37 zk-multiple-kafka-single.yml
-rw-rw-r-- 1 ec2-user ec2-user 2946 Nov 16 11:37 zk-single-kafka-multiple.yml
-rw-rw-r-- 1 ec2-user ec2-user 1604 Nov 16 11:37 zk-single-kafka-single.yml
Note the file zk-single-kafka-single.yml. As the name suggests, we'll use it to launch a Kafka cluster with a single Zookeeper and a single broker.
Now, use this command to launch a Kafka cluster with one Zookeeper and one Kafka broker. Add -d flag to run it in the background.
Check to make sure both the services are running:
1
2
3
4
5
$ docker-compose -f zk-single-kafka-single.yml ps
Name Command State Ports
-------------------------------------------------------------------------------------------
kafka1 /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp, 0.0.0.0:9999->9999/tcp
zoo1 /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcpKafka is going to be exposed on your computer at localhost:9092
To run Kafka commands against Docker, we have two options:
Run commands directly from within the Docker container of Kafka (using
docker exec)Run commands from our host OS (we must first install the binaries)
Option 1: Running commands from within the Kafka docker container
Then, from within the container, you can start running some Kafka commands (without .sh)
Option 2: Running commands from outside your container
For this, you must install the Java and the Kafka Binaries on your system:
instructions for Mac (follow the whole document except starting Kafka and Zookeeper)
instructions for Linux (follow the whole document except starting Kafka and Zookeeper)
instructions for Windows (follow the whole document except starting Kafka and Zookeeper)
One this is installed, the Kafka binaries should be in your path, and you will be good to issue your first Kafka command, for example kafka-topics.sh
Now that we've verified the installation, we can stop the services by stopping the respective containers:
1
2
3
$ docker-compose -f zk-single-kafka-single.yml stop
Stopping kafka1 ... done
Stopping zoo1 ... done To remove all the resources including the containers altogether, use down instead of stop:
Get a free Kafka cluster with Conduktor Platform
Conduktor Platform provides the easiest way to get started with Apache Kafka. Just sign up to our web app and you'll get a free managed Kafka cluster that you can use to learn and experiment. There is no trial period or constant pestering, just head to the signup page and get started
