reiseburo
/
hermann
mirror of https://github.com/reiseburo/hermann
1
0
Fork 0
Code Issues Releases Wiki Activity
304 Commits 9 Branches 20 Tags 604 KiB
Commit Graph

61 Commits

Author SHA1 Message Date
jakesandlund c64d38cff3 Comment that rdcrc32.h and rd_kafka_msg_partitioner_consistent can be removed when librdkafka tags and Hermann updates 2015-09-16 16:04:35 +00:00
cory and jakesandlund e8703e1df4 Add support for passing partition_key in MRI 2015-09-14 16:11:43 +00:00
Ben Osheroff 2c99af440e surround if with braces 2015-06-30 10:51:19 -07:00
Ben Osheroff 5b8dd6feef yield key and offset into Consumer#consume block 2015-06-30 10:49:37 -07:00
Ben Osheroff 9edc4b9301 move hermann_lib -> hermann_rdkafka / Hermann::Provider::RDKafka 2015-06-23 09:01:31 -07:00
Ben Osheroff a9d80242dd rename hermann_lib -> hermann_rdkafka 2015-06-22 19:39:56 -07:00
Ben Osheroff 783d7dac0d cleanup a few merge-induced warnings 2015-06-16 23:45:25 -07:00
Ben Osheroff 09df7ec1b9 Merge remote-tracking branch 'upstream/master' into cleanup_warnings 2015-06-16 23:44:13 -07:00
Ben Osheroff d8b8f83690 1.8.7 has no version.h 2015-06-16 20:55:55 -07:00
R. Tyler Croy 7f63e3c0d3 Merge pull request #109 from zendesk/better_postloop_cleanup 
Better postloop cleanup
 2015-06-15 06:55:35 -07:00
R. Tyler Croy c272bff063 Merge pull request #108 from zendesk/metadata 
Add the ability to request cluster / topic metadata from the brokers
 2015-06-15 06:54:31 -07:00
Ben Osheroff 74cba3c513 upgrade to librdkafka 0.8.6 
R. Tyler's patches are included in librdkafka 0.8.6
 2015-06-14 19:33:41 -07:00
Ben Osheroff 5c898144f2 cleanup warnings 
- fix partition selection function
- get ruby >= 2 calling the correct no-gvl function
 2015-06-13 19:47:07 -07:00
Ben Osheroff 17e5c5b31d ensure we call consumer_consume_loop_stop at the end of the loop 
we need rb_ensure so that if the loop terminates in a "break" statement
we'll still call rd_kafka_consume_stop
 2015-06-13 19:06:45 -07:00
Ben Osheroff 942fd87728 call rd_kafka_message_destroy right before rb_yield() 
if rb_yield() ends in a "break" statement, it never returns control back
to the caller; thus we leak the message.
 2015-06-13 19:05:25 -07:00
Ben Osheroff 0bc9e9d9ee don't destory metadata unless it's allocated 2015-06-12 18:11:34 -07:00
Ben Osheroff 5942e1810b fix bug, constantize TIMEOUT_MS, add README.md info 2015-05-19 11:33:13 -07:00
Ben Osheroff e38b626b71 Add metadata API to Hermann::Discovery module 2015-05-19 11:13:27 -07:00
Ben Osheroff efd1a80a37 1.9.3 fixes 
1.9.3 seems like it should actually still be using the
rb_thread_blocking_region() call, as ...call_without_gvl() isn't defined
in any header (although it is available).

Consuming that implicit declaration was causing crashes.
 2015-04-30 07:52:01 -07:00
Ben Osheroff 30669be4d5 Fix use-after-free bug 
it's not safe to simply pluck strings off the heap unless we were also
going to maintain a reference to them.  sidestep this problem by
strdup'ing the topic and brokers strings
 2015-04-29 14:33:58 -07:00
R. Tyler Croy 4aef448dbc Merge pull request #95 from zendesk/mri_offsets 
mri, initial offsets
 2015-04-28 15:41:33 -07:00
Ben Osheroff 4f4e3f9b55 add some documentation to the new MRI consumer functions 2015-04-28 11:07:33 -07:00
Ben Osheroff 8857531cb7 add a constant for the time we ignore the gvl's interrupt requests 2015-04-28 11:06:34 -07:00
Ben Osheroff 752123b83e allow choosing of an inital offset in the MRI consumer 2015-04-28 10:17:09 -07:00
Ben Osheroff 784a7f3afe 1.9.3 compatibility 2015-04-27 12:32:58 -07:00
Ben Osheroff d579f2f8ec use rb_thread_call_without_gvl 
this allows threaded code on modern (>=2.0) rubies working.  Also,
switch to the single-message form of the rd_kafka call -- the callback
form isn't safe to run without the GVL held, as it enters the ruby
interpreter in rb_yield().
 2015-04-21 14:02:48 -07:00
R. Tyler Croy f0b019ed69 Update the C provider to accept a topic into the consume method 
Note: this doesn't actually do anything just yet
 2014-10-27 09:42:01 -07:00
R. Tyler Croy 4261907955 Treat empty Content-Length headers as retryable errors for mitigating github.com brokeness 
Periodically we'll get 200 OK responses with not Content-Length header when
trying to download from github.com, this will treat it as a retry

Fixes #40
 2014-10-14 11:24:38 -07:00
R. Tyler Croy 249f2739a7 Properly propagate the optionally passed in topic on #push when recursing through Arrays 
This commit also moves tests back out of the platform specific scope and makes
the internal #push_single API more consistent to alllow easier testing.

Fixes #49
 2014-10-14 10:54:52 -07:00
R. Tyler Croy beb8c4764d Add a much more verbose download_file_http method for mini_portile 
This should help isolate the bug we're seeing in #40 which seem to stem from
GitHub serving invalid packages
 2014-10-14 09:53:42 -07:00
R. Tyler Croy 5b6fa5075f Support passing in a topic for every #push in MRI 
This commit also fixes a memory leak with the JRuby version of the gem where we
were not properly cleaning up children objects and holding onto references of
them forever

Fixes #46
 2014-10-14 09:51:24 -07:00
R. Tyler Croy 44efb9e2b5 Roll back to using 0.8.4 APIs since we're bundling now 2014-09-12 16:15:04 -07:00
R. Tyler Croy 5de9612882 Bundle a known good version of librdkafka (0.8.4) with Hermann 
This reduces the need for the librdkafka system dependency and results in
statically linking librdkafka.a against the C extension.

There's a few hacks in extconf.rb specifically to deal with librdkafka's build
system, as well as apply checksumming to the downloaded file

Fixes #8
 2014-09-12 16:13:19 -07:00
R. Tyler Croy afd14dec80 Revert back to using rd_kafka_conf_set_dr_cb() which is 0.8.3 compatible 
Fixes #36
 2014-09-12 14:24:53 -07:00
R. Tyler Croy 7af7e3ec5b Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly 
Fixes #32
 2014-09-10 20:00:07 -07:00
R. Tyler Croy 341e176b2f Propagate system broker errors into Hermann::Result objects 
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.

Fixes #25
 2014-09-10 11:04:14 -07:00
R. Tyler Croy 218c035793 Expose an #errored? method which gets set if the general error callback is invoked 
Fixes #25
 2014-09-09 17:37:05 -07:00
R. Tyler Croy 288653845a Introduce a wrapper struct to make message delivery callbacks able to change Producer data 
This will help with determining whether we're no longer connected due to
failures after a push of a message
 2014-09-09 16:45:27 -07:00
R. Tyler Croy 9014551d10 Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value 
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly

Fixes #25
 2014-09-09 14:40:10 -07:00
R. Tyler Croy 0f9681ab91 Hold onto the Hermann::Lib::Producer pointer inside of the rd_kafka_conf_t 2014-09-09 14:40:10 -07:00
R. Tyler Croy acec652d8b Switch out all these ifdef TRACE statements for a cleaner TRACER() macro 2014-09-09 14:40:10 -07:00
R. Tyler Croy 128ec66db0 Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls 
This also introduces some more/better error handling in Hermann::Lib::Producer#tick

Fixes #24
 2014-09-09 11:11:07 -07:00
R. Tyler Croy 08d1aab811 Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests 2014-09-09 10:51:49 -07:00
Stan Campbell 46d62a26cb Remove exit(..) calls and replace by raising a runtime exception to Ruby. 2014-09-09 09:19:28 -07:00
R. Tyler Croy 77248706e8 Use RSTRING_LEN instead of _LENINT which isn't present on 1.8.7 
Fixes #30
 2014-09-09 09:09:50 -07:00
R. Tyler Croy ed36300265 Pass a pointer to a Hermann::Result into the message delivery callback 
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.

What's still missing is producer level error calbacks

Fixes #11
Fixes #15
Fixes #17
 2014-09-04 14:31:29 -07:00
Stan Campbell 34193dd142 Merge pull request #22 from rtyler/issues/21-buffer-overrun 
Avoid double-ocpying and a buffer overrun when pushing a Ruby string
 2014-09-04 13:50:33 -07:00
Stan Campbell 3a48bd6f64 Merge pull request #20 from rtyler/issues/11-producer-feedback 
An initial pass at exposing the async nature of librdkafka upwards
 2014-09-04 13:50:14 -07:00
R. Tyler Croy 7085385d7e Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka 
This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely
copy the buffer for it's own uses from the Ruby string's C pointer.

This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21

Fixes #21
Fixes #13
 2014-09-04 13:03:01 -07:00
R. Tyler Croy 74dfd19399 Remove inadvertantly added changes to msg_delivered function 2014-09-04 11:55:19 -07:00