Using delivery_boy gem to Publish Messages to Kafka

Add the gem to Gemfile.

gem 'delivery_boy'

Run bundle. We can send a message synchronously.

pollywog@kafka:/src/messenger/rafk$ rails c
Running via Spring preloader in process 1642
Loading development environment (Rails 5.2.2)
irb(main):001:0> DeliveryBoy.deliver({message: 'Test message for comments'}.to_json, topic: "comments")
I, [2019-02-15T21:31:08.955191 #1642]  INFO -- : New topics added to target list: comments
I, [2019-02-15T21:31:08.956142 #1642]  INFO -- : Fetching cluster metadata from kafka://localhost:9092
D, [2019-02-15T21:31:08.956699 #1642] DEBUG -- : Opening connection to localhost:9092 with client id delivery_boy...
D, [2019-02-15T21:31:10.041469 #1642] DEBUG -- : Current leader for comments/0 is node localhost:9092 (node_id=0)
I, [2019-02-15T21:31:10.041864 #1642]  INFO -- : Sending 1 messages to localhost:9092 (node_id=0)
D, [2019-02-15T21:31:10.041979 #1642] DEBUG -- : Opening connection to localhost:9092 with client id delivery_boy...
D, [2019-02-15T21:31:10.043045 #1642] DEBUG -- : Sending produce API request 1 to localhost:9092
D, [2019-02-15T21:31:10.072703 #1642] DEBUG -- : Waiting for response 1 from localhost:9092
D, [2019-02-15T21:31:10.117206 #1642] DEBUG -- : Received response 1 from localhost:9092
D, [2019-02-15T21:31:10.117334 #1642] DEBUG -- : Successfully appended 1 messages to comments/0 on localhost:9092 (node_id=0)

We can read the message from the comments topic using Kafkacat.

pollywog@kafka:~$ kafkacat -b kafka -t comments
% Auto-selecting Consumer mode (use -P or -C to override)
{"message":"Test message for comments"}
% Reached end of topic comments [0] at offset 1

To free up the server process, we can deliver the message asynchronously that runs in its own process.

DeliveryBoy.deliver_async({message: 'Asynch delivery test'}.to_json, topic: "comments")

You can see the consumer output in the Kafkacat window.

 kafkacat -b kafka -t comments
% Auto-selecting Consumer mode (use -P or -C to override)
{"message":"Test message for comments"}
% Reached end of topic comments [0] at offset 1
{"message":"Asynch delivery test"}
% Reached end of topic comments [0] at offset 2

Waterdrop gem depends on delivery_boy gem. It is good to know how the delivery_boy works. It comes with a Rails generator to generate the configuration file.

pollywog@kafka:/src/messenger/rafk$ bundle exec rails g delivery_boy:install
Running via Spring preloader in process 1688
      create  config/delivery_boy.yml

The generated configuration file config/delivery_boy.yml.

# These config values will be shared by all environments but can be overridden.
common: &common
  client_id: "rafk"

development:
  <<: *common
  brokers:
    - localhost:9092

test:
  <<: *common
  brokers:
    - localhost:9092

production:
  <<: *common
  brokers:
    - kafka1.myapp.com:9092
    - kafka2.myapp.com:9092
    - kafka3.myapp.com:9092

You can deliver messages to a specific partition. For more details checkout the delivery_boy gem homepage.

References


Related Articles


Ace the Technical Interview

  • Easily find the gaps in your knowledge
  • Get customized lessons based on where you are
  • Take consistent action everyday
  • Builtin accountability to keep you on track
  • You will solve bigger problems over time
  • Get the job of your dreams

Take the 30 Day Coding Skills Challenge

Gain confidence to attend the interview

No spam ever. Unsubscribe anytime.