A Ruby gem wrapping the librdkafka library as a C extension.

Go to file

R. Tyler Croy 7085385d7e Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely copy the buffer for it's own uses from the Ruby string's C pointer. This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21 Fixes #21 Fixes #13		2014-09-04 13:03:01 -07:00
ext/hermann	Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka	2014-09-04 13:03:01 -07:00
lib	Major rework of the gemspec	2014-08-29 13:52:36 -07:00
scripts	Update the requires for the sample scripts.	2014-09-04 10:52:27 -07:00
spec	Properly raise an exception when the consumer is given an empty topic	2014-08-30 15:30:11 -07:00
.gitignore	Restructure the Hermann gem extension build process to use rake-compiler	2014-08-29 18:37:10 -07:00
.travis.yml	Added travis.yml	2014-08-18 13:37:46 -07:00
Gemfile	Restructure the Hermann gem extension build process to use rake-compiler	2014-08-29 18:37:10 -07:00
Gemfile.lock	Restructure the Hermann gem extension build process to use rake-compiler	2014-08-29 18:37:10 -07:00
LICENSE.md	Moved license to LICENSE.md and reformatted markdown. Closes #1	2014-08-18 13:48:56 -07:00
README.md	Add the gitter chat link to the readme	2014-09-02 14:01:55 -07:00
Rakefile	Depend on build instead of just compile in the default task	2014-08-29 18:37:41 -07:00
hermann.gemspec	Restructure the Hermann gem extension build process to use rake-compiler	2014-08-29 18:37:10 -07:00

README.md

Hermann

A Ruby gem implementing a Kafka Publisher and Consumer

This library wraps the librdkafka library which is implemented in C. This library must be installed before we can use the Hermann gem.

Usage

Usage is modelled on the kafka-rb gem and is fairly straightforward.

Kafka 0.8 is supported.
Ruby 1.8.7 and Ruby 2.1.1 have been tested, but JRuby and versions

= 1.8 should work as long as the librdkafka library is installed.
This is an early alpha version of the gem, so expect bumps in the road.

Note: The current implementation needs work to ensure thread safety.

Both Consumers and Producers now hold their own references to their Kafka state, and should be able to coexist nicely.

The brokers list and partition specification for Consumers are now working, as well. Producers will detect the number of partitions in a topic and randomly spread their messages over the set of partitions.

Consumers currently do not remember their "last message", nor do they yet coordinate partition (re)assigment within consumer groups.

Consumer

require 'hermann'

# Initialize requires topic, brokers list, and partition number

c = Hermann::Consumer.new( "device_actions", "localhost:9092", 0 )
c.consume() do
  |msg| puts("Received: #{msg}")
end

Producer

require 'hermann'

p = Hermann::Producer.new( "device_actions", "localhost:9092" )

messages = [ "Locate", "Scream", "Wipe", "Degauss" ]
messages.each do
    |msg| p.push(msg)
end

Questions?

Stan Campbell can be reached at stan.campbell3 at( @ ) gmail.com