Ruby Kafka Messaging App using Docker

Introduction

In the previous article, Kafka Producer and Consumer in Ruby using Docker we ran the producer and consumer programs in the Docker host.

In this article, we will connect a simple Ruby messaging app running in a container to Kafka running in another container. The docker-compose file is taken from an existing working example. The only change made is the version, 3.6 to the docker-compose.yml for single Kafka example.

If you don't have an empty Gemfile.lock, you will get the error:

ERROR: Service 'messenger' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 16
The deployment setting requires a Gemfile.lock. Please make sure you have checked your Gemfile.lock into version control before deploying.

Add an empty Gemfile.lock to the Dockerfile:

FROM ruby:2.5

# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1

WORKDIR /app
COPY . .
RUN bundle install 
CMD echo 'Messenger up...'

We can add the Ruby app to the docker-compose.yml and specify the build context for it to be the Dockerfile we created.

version: '3.6'

services:
  zoo1:
    image: zookeeper:3.4.9
    hostname: zoo1
    ports:
      - "2181:2181"
    environment:
        ZOO_MY_ID: 1
        ZOO_PORT: 2181
        ZOO_SERVERS: server.1=zoo1:2888:3888
    volumes:
      - ./zk-single-kafka-single/zoo1/data:/data
      - ./zk-single-kafka-single/zoo1/datalog:/datalog

  kafka1:
    image: confluentinc/cp-kafka:5.0.0
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./zk-single-kafka-single/kafka1/data:/var/lib/kafka/data
    depends_on:
      - zoo1
  messenger:
    build: .

Running docker-compose up gives the error:

Your bundle only supports platforms [] but your local platforms are ["ruby",
"x86_64-linux"], and there's no compatible match between those two lists.
ERROR: Service 'messenger' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 16

Adding the no-deployment option to the bundle install line:

bundle install --no-deployment

in the Dockerfile does not fix the problem. The cause is the bundle config line in the Dockerfile:

FROM ruby:2.5

# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1

WORKDIR /app

COPY Gemfile Gemfile.lock ./
RUN bundle install

COPY . .

CMD echo 'Messenger up...'

Remove that line and run docker-compose up again.

Create a producer.rb:

$LOAD_PATH.unshift(File.expand_path("../../lib", __FILE__))

require "kafka"

logger = Logger.new($stderr)
brokers = ENV.fetch("KAFKA_BROKERS")

# Make sure to create this topic in your Kafka cluster or configure the
# cluster to auto-create topics.
topic = "text"

kafka = Kafka.new(
  seed_brokers: brokers,
  client_id: "simple-producer",
  logger: logger,
)

producer = kafka.producer

begin
  $stdin.each do |line|
    producer.produce(line, topic: topic)

    producer.deliver_messages
  end
ensure
  producer.shutdown
end

The messenger ruby app will produce messages to Kafka. We can run the messenger Ruby app image:

$ docker run -it da215cfe53fd /bin/bash

This will run the image da215cfe53fd in a container.

The docker-compose does not recreate the ruby messenger image even if the Dockerfile is modified. To force the rebuild use the --build:

docker-compose up --build

To list only the names of the running containers.

$ docker ps --format '{{.Names}}'
messenger_kafka1_1
messenger_zoo1_1

To remove dangling images:

docker image prune -a

Running the example results in error:

kafka://kafka1:19092: getaddrinfo: Name or service not known

Go into the container and run bundle console:

kafka = Kafka.new(["kafka1:19092"], client_id: "my-application")
kafka.deliver_message("Hello, World!", topic: "greetings")

This does not work. The aha moment is that in order for Docker to connect the messenger container to the network where the kafka container is also connected, we need to have a process running. Add rack gem to Gemfile.

source 'https://rubygems.org'

gem 'rack'
gem 'ruby-kafka'

Create a simple Rack program hi.ru file that prints hello world.

run ->(env) { [200, {"Content-Type" => "text/html"}, ["Hello World!"]] }

Change CMD in Dockerfile to run hi.ru.

FROM ruby:2.5

WORKDIR /app
COPY . .
RUN bundle install

CMD ["rackup", "hi.ru"]

The KAFKA_ADVERTISED_LISTENERS configuration of the previous docker-compose.yml uses IP addresses. This can be problematic. Change the docker-compose.yml to use Confluent Docker Kafka image:

version: '3.6'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    # An important note about accessing Kafka from clients on other machines: 
    # -----------------------------------------------------------------------
    #
    # The config used here exposes port 9092 for _external_ connections to the broker
    # i.e. those from _outside_ the docker network. This could be from the host machine
    # running docker, or maybe further afield if you've got a more complicated setup. 
    # If the latter is true, you will need to change the value 'localhost' in 
    # KAFKA_ADVERTISED_LISTENERS to one that is resolvable to the docker host from those 
    # remote clients
    #
    # For connections _internal_ to the docker network, such as from other services
    # and components, use kafka:29092.
    #
    # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
    # "`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-'"`-._,-
    #
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  messenger:
    build: .

The KAFKA_ADVERTISED_LISTENERS is configured in a way that does not use IP. It allows the Docker host as well as another container to connect to Kafka instance. Rebuild image using docker-compose.

docker-compose up --build

We can now go into the messenger Ruby app container by providing the container id of the Ruby app:

docker exec -it cd989d5ca888 sh

Run bundle console. Produce a Hello World message as shown in the above example. We can now consume the message:

kafka.each_message(topic: "greetings") do |message|
  puts message.offset, message.key, message.value
end

It will work now. For testing purposes we can print only the message.

kafka.each_message(topic: "greetings") do |message|
  puts message.value
end

This time, we also see the messenger Ruby app running in a container.

docker ps

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                          NAMES
cd989d5ca888        messenger_messenger                "rackup hi.ru"           8 minutes ago       Up 8 minutes                                       messenger_messenger_1
31fd7033ebbc        confluentinc/cp-kafka:latest       "/etc/confluent/dock…"   3 hours ago         Up 8 minutes        0.0.0.0:9092->9092/tcp         messenger_kafka_1
cf4e9dad2bb0        confluentinc/cp-zookeeper:latest   "/etc/confluent/dock…"   3 hours ago         Up 8 minutes        2181/tcp, 2888/tcp, 3888/tcp   messenger_zookeeper_1

Running

docker network ls
NETWORK ID          NAME                   DRIVER              SCOPE
2ecc545ad2b8        alpy_default           bridge              local
a81a118ace21        c_default              bridge              local
b3c19c5fabc5        host                   host                local
5d3e7af46c8e        messenger_default      bridge              local

shows the messager_default network for our messenger Ruby app. All the containers based on the images in the docker-compose.yml are connected to this network. We can also test the connectivity by running the ping command.

$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                          NAMES
cd989d5ca888        messenger_messenger                "rackup hi.ru"           19 hours ago        Up 19 hours                                        messenger_messenger_1
31fd7033ebbc        confluentinc/cp-kafka:latest       "/etc/confluent/dock…"   23 hours ago        Up 19 hours         0.0.0.0:9092->9092/tcp         messenger_kafka_1
cf4e9dad2bb0        confluentinc/cp-zookeeper:latest   "/etc/confluent/dock…"   23 hours ago        Up 19 hours         2181/tcp, 2888/tcp, 3888/tcp   messenger_zookeeper_1

We can use the container ID from above to ssh into that container and executing the ping command.

$ docker exec -it cd989d5ca888 /bin/bash
root@cd989d5ca888:/app# ping kafka
PING kafka (172.29.0.4) 56(84) bytes of data.
64 bytes from messenger_kafka_1.messenger_default (172.29.0.4): icmp_seq=1 ttl=64 time=0.159 ms
64 bytes from messenger_kafka_1.messenger_default (172.29.0.4): icmp_seq=2 ttl=64 time=0.114 ms

In this case we are using the name of the service, kafka in the ping command. I found Julian's Docker Course very useful. Shoutout to Julian for answering Docker questions to get this article published.

References


Related Articles


Ace the Technical Interview

  • Easily find the gaps in your knowledge
  • Get customized lessons based on where you are
  • Take consistent action everyday
  • Builtin accountability to keep you on track
  • You will solve bigger problems over time
  • Get the job of your dreams

Take the 30 Day Coding Skills Challenge

Gain confidence to attend the interview

No spam ever. Unsubscribe anytime.