reiseburo
/
hermann
mirror of https://github.com/reiseburo/hermann
1
0
Fork 0
Code Issues Releases Wiki Activity
164 Commits 9 Branches 20 Tags 604 KiB
Commit Graph

50 Commits

Author SHA1 Message Date
R. Tyler Croy 228358edc1 Add a failing integration test for pumping protobufs through Hermann 2014-10-30 10:46:18 -07:00
R. Tyler Croy 7f4de34be0 Refactor the Hermann::Producer Java-based integration test to respect integration.yml 
I've also refactored/cleaned it up a bit to make sure it runs properly on my machine
 2014-10-30 10:46:18 -07:00
R. Tyler Croy 3c9c234c43 Update the Hermann::Discovery::Zookeeper API to return an Array of brokers more suitable for Hermann::Producer 2014-10-30 10:05:34 -07:00
jway 132dbb94a9 pass string to JavaProducer 2014-10-29 16:59:34 -07:00
R. Tyler Croy 4ea21d5add Account for a subtle internal API change between the C and Java Producers 2014-10-28 18:18:05 -07:00
jway 69202c21b2 allow integration test to run only in jruby 2014-10-28 11:14:14 -07:00
jway b48698035b update integration example with zookeepers 2014-10-28 08:35:33 -07:00
jway 9acce0a64b add shutdown 
* producer/consumer integration tests
 2014-10-27 21:16:17 -07:00
jway 5e0079ce05 allow passing topic into consume method 2014-10-25 09:12:47 -07:00
R. Tyler Croy 8c01040f5d Wrap calls to the underlying Java producer with some Ruby-based exception handling 
Fixes #58
 2014-10-24 11:47:51 -07:00
jway 972fe47410 add simple java consumer 2014-10-24 10:07:27 -07:00
R. Tyler Croy e0070b942b Properly reap children in tests on MRI and Java 
Fixes #54
 2014-10-15 12:21:29 -07:00
jway 63c64f8ea3 add reaping of java promises 2014-10-15 11:32:51 -07:00
jway 9ca7340b21 allow user to pass in opts 2014-10-14 15:17:54 -07:00
R. Tyler Croy 249f2739a7 Properly propagate the optionally passed in topic on #push when recursing through Arrays 
This commit also moves tests back out of the platform specific scope and makes
the internal #push_single API more consistent to alllow easier testing.

Fixes #49
 2014-10-14 10:54:52 -07:00
R. Tyler Croy 5b6fa5075f Support passing in a topic for every #push in MRI 
This commit also fixes a memory leak with the JRuby version of the gem where we
were not properly cleaning up children objects and holding onto references of
them forever

Fixes #46
 2014-10-14 09:51:24 -07:00
jway a6a1148221 allow java producer push to change topics 2014-10-13 15:17:10 -07:00
jway cd16650ad7 refactor zookeeper lookup into Discovery class 2014-10-10 11:12:22 -07:00
jway 354fbde7fa add zookeeper broker lookup to Java provider 2014-10-08 15:25:32 -07:00
R. Tyler Croy 4cfb92b697 Clean up the execution of RSpecs to run the platform-specific tests in the right place 2014-10-08 09:45:07 -07:00
jway 3151b3ef4b push method returns executing Promise 2014-10-07 15:27:36 -07:00
jway 17b6db78b1 updates specs, fixup producer exceptional handling 2014-10-06 11:23:32 -07:00
jway 07a1dcec8e add specs for java_producer 2014-10-06 08:08:30 -07:00
jway b898709042 first pass adding java kafka provider 2014-10-03 12:38:38 -07:00
R. Tyler Croy 7af7e3ec5b Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly 
Fixes #32
 2014-09-10 20:00:07 -07:00
R. Tyler Croy 254efad00a Add a failing integration test for binary data 
Just a quick reproduction case for #32
 2014-09-10 11:04:14 -07:00
R. Tyler Croy 341e176b2f Propagate system broker errors into Hermann::Result objects 
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.

Fixes #25
 2014-09-10 11:04:14 -07:00
R. Tyler Croy 218c035793 Expose an #errored? method which gets set if the general error callback is invoked 
Fixes #25
 2014-09-09 17:37:05 -07:00
R. Tyler Croy 288653845a Introduce a wrapper struct to make message delivery callbacks able to change Producer data 
This will help with determining whether we're no longer connected due to
failures after a push of a message
 2014-09-09 16:45:27 -07:00
R. Tyler Croy a0697f5b66 Refactor out the necessity for Hermann::Timeout and just rely in a Ruby-based busyloop 2014-09-09 16:45:17 -07:00
R. Tyler Croy 9014551d10 Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value 
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly

Fixes #25
 2014-09-09 14:40:10 -07:00
R. Tyler Croy 4d7e9f3b4e Add Hermann::Timeout abstraction for 1.8/1.9 compatible timeout support 2014-09-09 14:40:10 -07:00
R. Tyler Croy 128ec66db0 Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls 
This also introduces some more/better error handling in Hermann::Lib::Producer#tick

Fixes #24
 2014-09-09 11:11:07 -07:00
R. Tyler Croy 08d1aab811 Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests 2014-09-09 10:51:49 -07:00
R. Tyler Croy c2dfcf30f2 Add support for running integration tests through RSpec against a pre-configured broker/topic 
This will make it a bit easier to run local tests in a more automated fashion
against some non-public broker hostnames
 2014-09-09 10:47:22 -07:00
R. Tyler Croy 9bfd7ad2e6 Tie Hermann::Result#value to the underlying reactor to bring values up to Ruby 
This ensures that we're getting async values out of librdkafka into the calling
Ruby thread. Currently errors aren't being brought up properly, but we're getting there

Example:

    [14:47:30] tyler:Hermann git:(issues/11-producer-feedback*) $ pry -I lib -r 'hermann/producer'
    [1] pry(main)> p = Hermann::Producer.new('topic', 'kafka0.REDACTED.com:6667')
    => #<Hermann::Producer:0x00000803b3b450
    @brokers="kafka0.REDACTED.com:6667",
    @children=[],
    @internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
    @topic="topic">
    [2] pry(main)> r = p.push('hello world!')
    => #<Hermann::Result:0x00000803b8cb20
    @producer=
    #<Hermann::Producer:0x00000803b3b450
    @brokers="kafka0.REDACTED.com:6667",
    @children=[#<Hermann::Result:0x00000803b8cb20 ...>],
    @internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
    @topic="topic">,
    @reason=nil,
    @state=:unfulfilled,
    @value=nil>
    [3] pry(main)> r.state
    => :unfulfilled
    [4] pry(main)> r.value
    ticking rdkafka reactor
    ticked
    => "hello world!"
    [5] pry(main)> r.state
    => :fulfilled
    [6] pry(main)> r.rejected?
    => false
    [7] pry(main)>
    [14:47:56] tyler:Hermann git:(issues/11-producer-feedback*) $

Fixes #11
 2014-09-04 14:51:41 -07:00
R. Tyler Croy ed36300265 Pass a pointer to a Hermann::Result into the message delivery callback 
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.

What's still missing is producer level error calbacks

Fixes #11
Fixes #15
Fixes #17
 2014-09-04 14:31:29 -07:00
R. Tyler Croy ccc7d20d15 An initial pass at exposign the asynchronous nature of librdkafka up into Ruby 
This should be enough to get some initial review of the API going. Hooking up a
Hermann::Result class to the Hermann::Producer in Ruby is in this commit but
actually driving the librdkafka reactor underneath is not yet hooked up.

Fixes #11
Fixes #15
Fixes #18
 2014-09-03 14:19:09 -07:00
R. Tyler Croy 0eb18479b6 Properly raise an exception when the consumer is given an empty topic 2014-08-30 15:30:11 -07:00
R. Tyler Croy f5e0c9384d Raise an exception instead of attempting to write to an empty topic name 2014-08-30 13:21:33 -07:00
R. Tyler Croy 3effe046f4 Raise an exception instead of exiting process on bad brokers for Herman::Consumer 2014-08-30 13:21:10 -07:00
R. Tyler Croy f3bf216b66 Raise a RuntimeError when a bad partition is given to a consumer 
Fixes #7
 2014-08-30 12:54:53 -07:00
R. Tyler Croy d8166b7fe4 Raise a runtime error when a bad broker is specified 
Fixes #6
 2014-08-30 12:51:03 -07:00
R. Tyler Croy 4ca84f4ca9 Comment out tests which exit the Ruby interpreter until they can be fixed 2014-08-29 08:41:57 -07:00
R. Tyler Croy 603aba612a Implement enough of Herman::Consumer#consume to reproduce an exit(1) 2014-08-29 08:41:13 -07:00
R. Tyler Croy 941adaef3b Add test case that exits the ruby interpreter (to be fixed real soon) 2014-08-28 21:18:27 -07:00
R. Tyler Croy 164d87132b Default to documentation format 2014-08-28 21:17:44 -07:00
R. Tyler Croy ff300e4175 Start refactoring unnecessary logic out of the C-core, starting with Producer 2014-08-28 17:14:38 -07:00
R. Tyler Croy 96ff104061 Add some basic test cases to make sure that at least the library is getting loaded 2014-08-28 16:20:46 -07:00
R. Tyler Croy fbdf9b6055 Add rspec to the project for testing basic Hermann functionality 2014-08-28 16:13:12 -07:00