Prerequisite This tutorial assumes that Java 8 is installed. You check this with
1 | $ java -version |
1 2 3 | openjdk version "1.8.0_202" OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_202-b08) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.202-b08, mixed mode) |
If Java is not installed, you can install it on Mac with:
1 2 | $ brew tap AdoptOpenJDK/openjdk $ brew cask install adoptopenjdk8 |
Note: If you are using windows, use *.bat files instead of “*.ksh” in steps shown below. Replace “/” with “\” on the DOS terminal.
Step 1: Install Kafka in “/usr/local“. [on windows: c:/tools ]
1 2 3 4 5 6 | $ cd /usr/local $ sudo wget http://apache.mirror.amaze.com.au/kafka/2.1.0/kafka_2.11-2.1.0.tgz $ sudo tar -xzvf kafka_2.11-2.1.0.tgz $ sudo -R chgrp john:admin kafka_2.11-2.1.0 #john is the user & admin is the group $ cd kafka_2.11-2.1.0 |
Kafka & internal zookeeper will be installed in the folder “/usr/local/kafka_2.11-2.1.0”
Step 2: Start zookeeper from “/usr/local/kafka_2.11-2.1.0”.
1 2 | /usr/local/kafka_2.11-2.1.0]$ ./bin/zookeeper-server-start.sh ./config/zookeeper.properties |
The zookeeper will be running on “localhost:2181”
Step 3: Open a new terminal window and start kafka.
1 2 3 | $ cd /usr/local/kafka_2.11-2.1.0 /usr/local/kafka_2.11-2.1.0]$ ./bin/kafka-server-start.sh ./config/server.properties |
The Kafka server will be running on “localhost:9092“. You can verify this with
1 2 3 4 | telnet localhost 9092 Trying ::1... Connected to localhost. Escape character is '^]'. |
Step 4: Open a new terminal window and create a new topic named “test”.
1 2 3 | $ cd /usr/local/kafka_2.11-2.1.0 /usr/local/kafka_2.11-2.1.0]$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test |
Step 5: Open a new terminal window and create a producer.
1 2 3 4 | $ cd /usr/local/kafka_2.11-2.1.0 /usr/local/kafka_2.11-2.1.0]$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test > |
You will get a prompt “>” where you can type messages to be sent to the Kafka topic.
Step 6: Open a new terminal window and create a consumer.
1 2 3 | $ cd /usr/local/kafka_2.11-2.1.0 /usr/local/kafka_2.11-2.1.0]$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning |
Send & Receive messages
Now if you send any text message from the terminal where producer was started, you will see that message in the consumer terminal.
For example:
Producer:
1 2 3 4 | >test message 1 >test message 2 > |
Consumer:
1 2 3 | test message 1 test message 2 |
