Kafka 1.0 Basics

Introduction

If you are familiar with CAP theorem, Kafka is optimized for Consistency and Availability. It provides parallelism and decoupling. In this article, we will install Kafka and produce and consume messages using the shell scripts shipped with Kafka.

Step 1 : Install Ubuntu 16.04

I used Vagrant to install Kafka.

vagrant init ubuntu/xenial64

Install Ubuntu.

vagrant up

Customize Vagrantfile to bump up the memory.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"
  config.vm.network "private_network", ip: "192.168.33.10"
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end  
end

Make Vagrant pickup the config changes.

vagrant reload

Step 2 : Install Java 8

I ran into issue when installing Java 9.

E: Package 'oracle-java9-installer' has no installation candidate

SSH into the Ubuntu.

vagrant ssh

Run:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

You can check the installation.

javac -version

Output: javac 1.8.0_191

Step 3 : Install Zookeeper

Run:

sudo apt-get install zookeeperd

Check if Zookeeper is automatically started.

vagrant@ubuntu-xenial:~$ netstat -ant | grep :2181
tcp6       0      0 :::2181                 :::*                    LISTEN   

Step 4 : Install Kafka

I am not installing the latest Kafka version. I want to stick with version that is widely supported by ruby libraries to interact with Kafka. Download the binary.

wget https://archive.apache.org/dist/kafka/1.0.2/kafka_2.11-1.0.2.tgz

Extract the tar file.

tar -xzf kafka_2.11-1.0.2.tgz

Step 5 : Produce and Consume Messages

Start the server.

cd kafka_2.11-1.0.2/
bin/kafka-server-start.sh config/server.properties &

Create a topic called test.

/temp/kafka_2.11-1.0.2$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
[2018-11-24 21:07:01,560] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions test-0 (kafka.server.ReplicaFetcherManager)
[2018-11-24 21:07:01,588] INFO [Log partition=test-0, dir=/tmp/kafka-logs] Loading producer state from offset 0 with message format version 2 (kafka.log.Log)
[2018-11-24 21:07:01,593] INFO [Log partition=test-0, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 16 ms (kafka.log.Log)
[2018-11-24 21:07:01,595] INFO Created log for partition [test,0] in /tmp/kafka-logs with properties {compression.type -> producer, message.format.version -> 1.0-IV0, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2018-11-24 21:07:01,596] INFO [Partition test-0 broker=0] No checkpointed highwatermark is found for partition test-0 (kafka.cluster.Partition)
[2018-11-24 21:07:01,598] INFO Replica loaded for partition test-0 with initial high watermark 0 (kafka.cluster.Replica)
[2018-11-24 21:07:01,599] INFO [Partition test-0 broker=0] test-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)

You can list the topics:

bin/kafka-topics.sh --list --zookeeper localhost:2181

Let's send some messages to the topic by running the producer.

vagrant@ubuntu-xenial:~/temp/kafka_2.11-1.0.2$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> Hello Kafka
> [2018-11-24 21:08:36,642] INFO Updated PartitionLeaderEpoch. New: {epoch:0, offset:0}, Current: {epoch:-1, offset-1} for Partition: test-0. Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache)
First Kafka Test

You can hit control+C to kill the producer. Start the consumer to read from the test topic.

vagrant@ubuntu-xenial:~/temp/kafka_2.11-1.0.2$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

 Hello Kafka
 First Kafka Test

You can see that the consumer read the messages in the test topic and printed it to the standard output.

References

Vagrant Virtualbox Nodejs and Featherjs
Installing Kafka and Zookeeper on Vagrant Ubuntu 16.04
Oracle-Java8-Installer: No installation candidate
Kafka Quickstart
Kafka Downloads


Related Articles


Ace the Technical Interview

  • Easily find the gaps in your knowledge
  • Get customized lessons based on where you are
  • Take consistent action everyday
  • Builtin accountability to keep you on track
  • You will solve bigger problems over time
  • Get the job of your dreams

Take the 30 Day Coding Skills Challenge

Gain confidence to attend the interview

No spam ever. Unsubscribe anytime.