f0b019ed69
Update the C provider to accept a topic into the consume method
...
Note: this doesn't actually do anything just yet
2014-10-27 09:42:01 -07:00
4261907955
Treat empty Content-Length headers as retryable errors for mitigating github.com brokeness
...
Periodically we'll get 200 OK responses with not Content-Length header when
trying to download from github.com, this will treat it as a retry
Fixes #40
2014-10-14 11:24:38 -07:00
249f2739a7
Properly propagate the optionally passed in topic on #push when recursing through Arrays
...
This commit also moves tests back out of the platform specific scope and makes
the internal #push_single API more consistent to alllow easier testing.
Fixes #49
2014-10-14 10:54:52 -07:00
beb8c4764d
Add a much more verbose download_file_http method for mini_portile
...
This should help isolate the bug we're seeing in #40 which seem to stem from
GitHub serving invalid packages
2014-10-14 09:53:42 -07:00
5b6fa5075f
Support passing in a topic for every #push in MRI
...
This commit also fixes a memory leak with the JRuby version of the gem where we
were not properly cleaning up children objects and holding onto references of
them forever
Fixes #46
2014-10-14 09:51:24 -07:00
44efb9e2b5
Roll back to using 0.8.4 APIs since we're bundling now
2014-09-12 16:15:04 -07:00
5de9612882
Bundle a known good version of librdkafka (0.8.4) with Hermann
...
This reduces the need for the librdkafka system dependency and results in
statically linking librdkafka.a against the C extension.
There's a few hacks in extconf.rb specifically to deal with librdkafka's build
system, as well as apply checksumming to the downloaded file
Fixes #8
2014-09-12 16:13:19 -07:00
afd14dec80
Revert back to using rd_kafka_conf_set_dr_cb() which is 0.8.3 compatible
...
Fixes #36
2014-09-12 14:24:53 -07:00
7af7e3ec5b
Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly
...
Fixes #32
2014-09-10 20:00:07 -07:00
341e176b2f
Propagate system broker errors into Hermann::Result objects
...
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.
Fixes #25
2014-09-10 11:04:14 -07:00
218c035793
Expose an #errored? method which gets set if the general error callback is invoked
...
Fixes #25
2014-09-09 17:37:05 -07:00
288653845a
Introduce a wrapper struct to make message delivery callbacks able to change Producer data
...
This will help with determining whether we're no longer connected due to
failures after a push of a message
2014-09-09 16:45:27 -07:00
9014551d10
Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value
...
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly
Fixes #25
2014-09-09 14:40:10 -07:00
0f9681ab91
Hold onto the Hermann::Lib::Producer pointer inside of the rd_kafka_conf_t
2014-09-09 14:40:10 -07:00
acec652d8b
Switch out all these ifdef TRACE statements for a cleaner TRACER() macro
2014-09-09 14:40:10 -07:00
128ec66db0
Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls
...
This also introduces some more/better error handling in Hermann::Lib::Producer#tick
Fixes #24
2014-09-09 11:11:07 -07:00
08d1aab811
Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests
2014-09-09 10:51:49 -07:00
46d62a26cb
Remove exit(..) calls and replace by raising a runtime exception to Ruby.
2014-09-09 09:19:28 -07:00
77248706e8
Use RSTRING_LEN instead of _LENINT which isn't present on 1.8.7
...
Fixes #30
2014-09-09 09:09:50 -07:00
ed36300265
Pass a pointer to a Hermann::Result into the message delivery callback
...
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.
What's still missing is producer level error calbacks
Fixes #11
Fixes #15
Fixes #17
2014-09-04 14:31:29 -07:00
34193dd142
Merge pull request #22 from rtyler/issues/21-buffer-overrun
...
Avoid double-ocpying and a buffer overrun when pushing a Ruby string
2014-09-04 13:50:33 -07:00
3a48bd6f64
Merge pull request #20 from rtyler/issues/11-producer-feedback
...
An initial pass at exposing the async nature of librdkafka upwards
2014-09-04 13:50:14 -07:00
7085385d7e
Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka
...
This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely
copy the buffer for it's own uses from the Ruby string's C pointer.
This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21
Fixes #21
Fixes #13
2014-09-04 13:03:01 -07:00
74dfd19399
Remove inadvertantly added changes to msg_delivered function
2014-09-04 11:55:19 -07:00
e71ab6cd14
Update the requires for the sample scripts.
2014-09-04 10:52:27 -07:00
ccc7d20d15
An initial pass at exposign the asynchronous nature of librdkafka up into Ruby
...
This should be enough to get some initial review of the API going. Hooking up a
Hermann::Result class to the Hermann::Producer in Ruby is in this commit but
actually driving the librdkafka reactor underneath is not yet hooked up.
Fixes #11
Fixes #15
Fixes #18
2014-09-03 14:19:09 -07:00
0eb18479b6
Properly raise an exception when the consumer is given an empty topic
2014-08-30 15:30:11 -07:00
655cc4463a
Fix almost all of the compiler warnings
...
Some are intentionally left out of the fixing because I believe there's a
failing test case that needs to be added
2014-08-30 15:30:11 -07:00
f5e0c9384d
Raise an exception instead of attempting to write to an empty topic name
2014-08-30 13:21:33 -07:00
3effe046f4
Raise an exception instead of exiting process on bad brokers for Herman::Consumer
2014-08-30 13:21:10 -07:00
2f5e9ae4e3
Make a plethora of whitespace and style consistency fixes to C code
2014-08-30 13:06:19 -07:00
f3bf216b66
Raise a RuntimeError when a bad partition is given to a consumer
...
Fixes #7
2014-08-30 12:54:53 -07:00
d8166b7fe4
Raise a runtime error when a bad broker is specified
...
Fixes #6
2014-08-30 12:51:03 -07:00
3a6a54def6
C code should be formatted with tabs not spaces
2014-08-30 08:40:58 -07:00
e241f086ee
Restructure the Hermann gem extension build process to use rake-compiler
...
This is a little bit better and cleaner IMO than the hand-crafted rake tasks.
Would you like to know more? <https://github.com/luislavena/rake-compiler >
2014-08-29 18:37:10 -07:00
2f194b4352
Nuke ruby-version
2014-08-28 21:15:43 -07:00
ff300e4175
Start refactoring unnecessary logic out of the C-core, starting with Producer
2014-08-28 17:14:38 -07:00
ae0b964934
Turn off TRACE and DEBUG by default.
...
JIRA: PART-3886
2014-08-19 13:37:22 -07:00
1185d54ff1
Add bulletproofing around the management of config->rk and config->rkt kafka handles.
...
Avoid dereferencing or freeing null handles for Consumer and Producer classes.
JIRA: PART-3886
2014-08-19 13:32:32 -07:00
a18272e901
Remove Makefile and mkmf.log as they're generated by extconf.rb and mkmf.
2014-07-25 12:52:39 -07:00
80754f6296
Remove bogus binaries from commit.
2014-07-22 11:37:41 -07:00
9c04fb1bc1
Add call to rb_thread_blocking_region to release the Ruby GVM lock and allow other threads to execute while we await IO from Kafka.
2014-06-20 10:52:50 -07:00
d9c2a93ec8
Add partition and brokers list configuration to the constructors for both Consumer and Producer.
2014-06-16 11:12:44 -07:00
aed11dcfac
Update documentation and gem version readying for release.
2014-06-13 10:47:34 -07:00
bcccda15a3
Removing more printf's. Removed the UUID to reduce dependencies.
2014-06-06 13:29:38 -07:00
874f02e673
Removed some extraneous logging. A little code cleanup.
2014-06-06 12:25:03 -07:00
6f0779d1b7
Update producer to use higher performance (don't wait so long in rd_kafka_poll).
2014-06-05 16:22:55 -07:00
d7c6387d21
Change the consumer code to use the batched callback method rather than blocking and timing out on single messages.
2014-06-05 16:03:09 -07:00
691d591d8c
Code cleanup, added a UUID to the configuration per instance for debugging.
2014-06-05 15:33:08 -07:00
a7a35deda9
All state now in configuration associated with instance. However, signal callbacks for trapping exit don't have the proper signature
2014-06-05 09:28:01 -07:00
R. Tyler Croy
Stan Campbell
Stan Campbell