You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Go to file
R. Tyler Croy d822ae541a
Merge pull request #138 from bitwiseman/patch-3
6 years ago
ext/hermann Comment that rdcrc32.h and rd_kafka_msg_partitioner_consistent can be removed when librdkafka tags and Hermann updates 8 years ago
lib depend on concurrent-ruby ~> 1.0.0 7 years ago
scripts update api to have #topic(topic) as well as #topics 8 years ago
spec Add some code coverage while we're here 7 years ago
.gitignore Add some code coverage while we're here 7 years ago
.travis.yml No sense supporting ree in any capacity any more 7 years ago
Gemfile Locked rake version to ~11.3.0 6 years ago Update with mri instructions 8 years ago
Jarfile Bring back Jarfile and remove curator from dependency 8 years ago
Jenkinsfile Updated for latest Pipeline fixes 6 years ago Moved license to and reformatted markdown. Closes #1 9 years ago Update 7 years ago Document the current janky release process 8 years ago
Rakefile Add support for generating JUnit-compatible output for Jenkins 7 years ago
hermann.gemspec depend on concurrent-ruby ~> 1.0.0 7 years ago


Gitter chat Build Status

A Ruby gem implementing a Kafka Publisher and Consumer

On MRI (C-based Ruby), this library wraps the librdkafka library which is implemented in C.

On JRuby this library declares jar dependencies inside the .gemspec to express dependencies on the Java-based Kafka library provided by the Kafka project. Tools like jbundler will handle these declarations correctly.


Usage is modelled on the kafka-rb gem and is fairly straightforward.

  • Kafka 0.8 is supported.
  • Ruby 1.9.3, 2.1.1 and JRuby are tested against


Zookeeper discovery

Discover Kafka brokers through zookeeper. Looks at /brokers/ids in Zookeeper to find the list of brokers.

require 'hermann/producer'
require 'hermann/discovery/zookeeper'

broker_ids_array ='localhost:2181').get_brokers
producer ='topic', broker_ids_array)

promise = producer.push('hello world') # send message to kafka
promise.value                          # forces the Concurrent::Promise to finish excuting (#value!)
promise.state                          # the state of the promise

MRI only

require 'hermann/producer'

broker_ids_array ='localhost:2181').get_brokers
p ='topic', broker_ids_array)  # arguments topic, list of brokers
f = p.push('hello world from mri')


Messages can be consumed by calling the consume method and passing a block to handle the yielded messages. The consume method blocks, so take care to handle that functionality appropriately (i.e. use Concurrent::Promise, Thread, etc).


require 'hermann'
require 'hermann/consumer'
require 'hermann_jars'

topic     = 'topic'
new_topic = 'other_topic'

the_consumer =, zookeepers: "localhost:2181", group_id: "group1")

the_consumer.consume(new_topic) do |msg|   # can change topic with optional argument to .consume
  puts "Recv: #{msg}"


MRI currently has no zookeeper / client group support.

require 'hermann'
require 'hermann/consumer'

topic     = 'topic'
new_topic = 'other_topic'

the_consumer =, brokers: "localhost:9092", partition: 1)

the_consumer.consume(new_topic) do |msg, key, offset|   # can change topic with optional argument to .consume
  puts "Recv: #{msg}, key: #{key}, offset: #{offset}"

Metadata request (MRI-only)

Topic and cluster metadata may be retrieved in the MRI version by querying the Kafka brokers.

require 'hermann'
require 'hermann/discovery/metadata'

c = "localhost:9092" )
topic = c.topic("topic")

puts topic.partitions.first

consumers = do |partition|

Build & Unit Test

First time (from a clean repository): bundle install && bundle exec rake

Thereafter: bundle exec rake spec


To run the integration tests:

  • startup your own instance of zookeeper/kafka
  • rspec spec/integration/producer_spec.rb

How to convert from using jruby-kafka

  • Gemfile
    • remove jruby-kafka
    • add gem "hermann"
    • bundle install
  • Jarfile
    • removed unecessary jars from your Jarfile (i.e. kafka, log4j)
    • jar dependencies are automatically included with Hermann
    • jbundle install
  • Test out one of the Producer/Consumer examples above