reiseburo
/
hermann
mirror of https://github.com/reiseburo/hermann
1
0
Fork 0
Code Issues Releases Wiki Activity
164 Commits 9 Branches 20 Tags 604 KiB
Commit Graph

164 Commits

Author SHA1 Message Date
R. Tyler Croy 23e4cb41d9 Merge pull request #43 from rtyler/platform-specific-specs 
Clean up RSpecs to run the platform-specific tests in the right runtime
 2014-10-08 11:07:04 -07:00
R. Tyler Croy 32bb2f92bc Remove support for Ruby 1.8.7 
It's like, a million times dead. Concurrent Ruby won't support it which forces our hand though.
 2014-10-08 10:40:59 -07:00
R. Tyler Croy d2d99c1e0b Avoid building the C extension on JRuby 2014-10-08 10:10:08 -07:00
R. Tyler Croy c6779d8f56 Add JRuby to the list of rubies we should test on 2014-10-08 09:54:19 -07:00
R. Tyler Croy 4cfb92b697 Clean up the execution of RSpecs to run the platform-specific tests in the right place 2014-10-08 09:45:07 -07:00
R. Tyler Croy 05a147444d Remove Gemfile.lock which will now be dependent on the version of Ruby 
The contents will be wildly different depending on an MRI vs. JRuby-based installation
 2014-10-07 17:39:12 -07:00
R. Tyler Croy 287e350161 Merge pull request #42 from lookout/execute_promise 
Push method returns executing Promise
 2014-10-07 15:41:20 -07:00
jway 3151b3ef4b push method returns executing Promise 2014-10-07 15:27:36 -07:00
R. Tyler Croy a08cfaa155 Merge pull request #41 from jamescway/add-jruby-producer 
Java Producer Provider
 2014-10-06 16:07:08 -07:00
jway 06dda845b8 add rdoc 2014-10-06 13:46:00 -07:00
jway 17b6db78b1 updates specs, fixup producer exceptional handling 2014-10-06 11:23:32 -07:00
jway de03fa4ebc add require jars to top level module 2014-10-06 09:33:35 -07:00
jway 07a1dcec8e add specs for java_producer 2014-10-06 08:08:30 -07:00
jway b898709042 first pass adding java kafka provider 2014-10-03 12:38:38 -07:00
Stan Campbell f1ef2772e2 Merge pull request #38 from rtyler/issues/8-bundle-librdkafka 
Bundle librdkafka 0.8.4 with the gem
 2014-09-12 16:32:43 -07:00
R. Tyler Croy 7bf9cf745d Disable rubinius builds in travis 2014-09-12 16:26:37 -07:00
R. Tyler Croy 44efb9e2b5 Roll back to using 0.8.4 APIs since we're bundling now 2014-09-12 16:15:04 -07:00
R. Tyler Croy 5de9612882 Bundle a known good version of librdkafka (0.8.4) with Hermann 
This reduces the need for the librdkafka system dependency and results in
statically linking librdkafka.a against the C extension.

There's a few hacks in extconf.rb specifically to deal with librdkafka's build
system, as well as apply checksumming to the downloaded file

Fixes #8
 2014-09-12 16:13:19 -07:00
R. Tyler Croy afd14dec80 Revert back to using rd_kafka_conf_set_dr_cb() which is 0.8.3 compatible 
Fixes #36
 2014-09-12 14:24:53 -07:00
R. Tyler Croy d8f72fc9f2 Bump to 0.16.0 for the next release 2014-09-12 14:12:53 -07:00
Stan Campbell e7f0f900ad Merge pull request #35 from rtyler/issues/32-binary-data 
Avoid call to rb_string_value_cstr() which doesn't handle binary strings properly
 2014-09-11 11:34:24 -07:00
R. Tyler Croy 7af7e3ec5b Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly 
Fixes #32
 2014-09-10 20:00:07 -07:00
Stan Campbell 838904a035 Merge pull request #33 from rtyler/issues/25-producer-errors 
Propagate systemic errors from librdkafka up into Herman::Result objects
 2014-09-10 12:48:07 -07:00
R. Tyler Croy 254efad00a Add a failing integration test for binary data 
Just a quick reproduction case for #32
 2014-09-10 11:04:14 -07:00
R. Tyler Croy 341e176b2f Propagate system broker errors into Hermann::Result objects 
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.

Fixes #25
 2014-09-10 11:04:14 -07:00
Stan Campbell cea9161bbb Merge pull request #31 from rtyler/issues/25-producer-errors 
Add lots more machinery to help identify errors outside of a `push` command
 2014-09-10 10:26:50 -07:00
R. Tyler Croy 218c035793 Expose an #errored? method which gets set if the general error callback is invoked 
Fixes #25
 2014-09-09 17:37:05 -07:00
R. Tyler Croy 288653845a Introduce a wrapper struct to make message delivery callbacks able to change Producer data 
This will help with determining whether we're no longer connected due to
failures after a push of a message
 2014-09-09 16:45:27 -07:00
R. Tyler Croy a0697f5b66 Refactor out the necessity for Hermann::Timeout and just rely in a Ruby-based busyloop 2014-09-09 16:45:17 -07:00
R. Tyler Croy 9014551d10 Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value 
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly

Fixes #25
 2014-09-09 14:40:10 -07:00
R. Tyler Croy 4d7e9f3b4e Add Hermann::Timeout abstraction for 1.8/1.9 compatible timeout support 2014-09-09 14:40:10 -07:00
R. Tyler Croy 0f9681ab91 Hold onto the Hermann::Lib::Producer pointer inside of the rd_kafka_conf_t 2014-09-09 14:40:10 -07:00
R. Tyler Croy acec652d8b Switch out all these ifdef TRACE statements for a cleaner TRACER() macro 2014-09-09 14:40:10 -07:00
R. Tyler Croy 08f47503ed Add a Rake task for running integration tests 2014-09-09 11:18:08 -07:00
R. Tyler Croy 128ec66db0 Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls 
This also introduces some more/better error handling in Hermann::Lib::Producer#tick

Fixes #24
 2014-09-09 11:11:07 -07:00
R. Tyler Croy 08d1aab811 Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests 2014-09-09 10:51:49 -07:00
R. Tyler Croy c2dfcf30f2 Add support for running integration tests through RSpec against a pre-configured broker/topic 
This will make it a bit easier to run local tests in a more automated fashion
against some non-public broker hostnames
 2014-09-09 10:47:22 -07:00
R. Tyler Croy 99eb52e781 Add pry to the gemfile for better local testing/dev 2014-09-09 10:45:55 -07:00
Stan Campbell 46d62a26cb Remove exit(..) calls and replace by raising a runtime exception to Ruby. 2014-09-09 09:19:28 -07:00
R. Tyler Croy 77248706e8 Use RSTRING_LEN instead of _LENINT which isn't present on 1.8.7 
Fixes #30
 2014-09-09 09:09:50 -07:00
Stan Campbell 5cf9f352e1 Merge pull request #23 from rtyler/issues/11-producer-feedback 
Tie Hermann::Result to the librdkafka reactor
 2014-09-04 15:09:44 -07:00
R. Tyler Croy 9bfd7ad2e6 Tie Hermann::Result#value to the underlying reactor to bring values up to Ruby 
This ensures that we're getting async values out of librdkafka into the calling
Ruby thread. Currently errors aren't being brought up properly, but we're getting there

Example:

    [14:47:30] tyler:Hermann git:(issues/11-producer-feedback*) $ pry -I lib -r 'hermann/producer'
    [1] pry(main)> p = Hermann::Producer.new('topic', 'kafka0.REDACTED.com:6667')
    => #<Hermann::Producer:0x00000803b3b450
    @brokers="kafka0.REDACTED.com:6667",
    @children=[],
    @internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
    @topic="topic">
    [2] pry(main)> r = p.push('hello world!')
    => #<Hermann::Result:0x00000803b8cb20
    @producer=
    #<Hermann::Producer:0x00000803b3b450
    @brokers="kafka0.REDACTED.com:6667",
    @children=[#<Hermann::Result:0x00000803b8cb20 ...>],
    @internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
    @topic="topic">,
    @reason=nil,
    @state=:unfulfilled,
    @value=nil>
    [3] pry(main)> r.state
    => :unfulfilled
    [4] pry(main)> r.value
    ticking rdkafka reactor
    ticked
    => "hello world!"
    [5] pry(main)> r.state
    => :fulfilled
    [6] pry(main)> r.rejected?
    => false
    [7] pry(main)>
    [14:47:56] tyler:Hermann git:(issues/11-producer-feedback*) $

Fixes #11
 2014-09-04 14:51:41 -07:00
R. Tyler Croy ed36300265 Pass a pointer to a Hermann::Result into the message delivery callback 
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.

What's still missing is producer level error calbacks

Fixes #11
Fixes #15
Fixes #17
 2014-09-04 14:31:29 -07:00
R. Tyler Croy 3a3df235a7 Properly pass the block to Consumer#consume into the C layer 2014-09-04 14:29:54 -07:00
R. Tyler Croy dc9a057eaa Add a simple script to validate big message behavior 
References #21
 2014-09-04 14:29:29 -07:00
Stan Campbell 34193dd142 Merge pull request #22 from rtyler/issues/21-buffer-overrun 
Avoid double-ocpying and a buffer overrun when pushing a Ruby string
 2014-09-04 13:50:33 -07:00
Stan Campbell 3a48bd6f64 Merge pull request #20 from rtyler/issues/11-producer-feedback 
An initial pass at exposing the async nature of librdkafka upwards
 2014-09-04 13:50:14 -07:00
R. Tyler Croy 7085385d7e Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka 
This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely
copy the buffer for it's own uses from the Ruby string's C pointer.

This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21

Fixes #21
Fixes #13
 2014-09-04 13:03:01 -07:00
R. Tyler Croy 74dfd19399 Remove inadvertantly added changes to msg_delivered function 2014-09-04 11:55:19 -07:00
Stan Campbell e71ab6cd14 Update the requires for the sample scripts. 2014-09-04 10:52:27 -07:00