Stan Campbell
c0f3f5bf51
Remove TRACE flag left on in last commit.
2014-09-23 12:41:44 -07:00
Stan Campbell
32c97deee2
Temporarily disable the caching of Result objects at the Producer.
2014-09-23 12:11:09 -07:00
R. Tyler Croy
44efb9e2b5
Roll back to using 0.8.4 APIs since we're bundling now
2014-09-12 16:15:04 -07:00
R. Tyler Croy
5de9612882
Bundle a known good version of librdkafka (0.8.4) with Hermann
...
This reduces the need for the librdkafka system dependency and results in
statically linking librdkafka.a against the C extension.
There's a few hacks in extconf.rb specifically to deal with librdkafka's build
system, as well as apply checksumming to the downloaded file
Fixes #8
2014-09-12 16:13:19 -07:00
R. Tyler Croy
afd14dec80
Revert back to using rd_kafka_conf_set_dr_cb() which is 0.8.3 compatible
...
Fixes #36
2014-09-12 14:24:53 -07:00
R. Tyler Croy
7af7e3ec5b
Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly
...
Fixes #32
2014-09-10 20:00:07 -07:00
R. Tyler Croy
341e176b2f
Propagate system broker errors into Hermann::Result objects
...
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.
Fixes #25
2014-09-10 11:04:14 -07:00
R. Tyler Croy
218c035793
Expose an #errored? method which gets set if the general error callback is invoked
...
Fixes #25
2014-09-09 17:37:05 -07:00
R. Tyler Croy
288653845a
Introduce a wrapper struct to make message delivery callbacks able to change Producer data
...
This will help with determining whether we're no longer connected due to
failures after a push of a message
2014-09-09 16:45:27 -07:00
R. Tyler Croy
9014551d10
Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value
...
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly
Fixes #25
2014-09-09 14:40:10 -07:00
R. Tyler Croy
0f9681ab91
Hold onto the Hermann::Lib::Producer pointer inside of the rd_kafka_conf_t
2014-09-09 14:40:10 -07:00
R. Tyler Croy
acec652d8b
Switch out all these ifdef TRACE statements for a cleaner TRACER() macro
2014-09-09 14:40:10 -07:00
R. Tyler Croy
128ec66db0
Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls
...
This also introduces some more/better error handling in Hermann::Lib::Producer#tick
Fixes #24
2014-09-09 11:11:07 -07:00
R. Tyler Croy
08d1aab811
Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests
2014-09-09 10:51:49 -07:00
Stan Campbell
46d62a26cb
Remove exit(..) calls and replace by raising a runtime exception to Ruby.
2014-09-09 09:19:28 -07:00
R. Tyler Croy
77248706e8
Use RSTRING_LEN instead of _LENINT which isn't present on 1.8.7
...
Fixes #30
2014-09-09 09:09:50 -07:00
R. Tyler Croy
ed36300265
Pass a pointer to a Hermann::Result into the message delivery callback
...
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.
What's still missing is producer level error calbacks
Fixes #11
Fixes #15
Fixes #17
2014-09-04 14:31:29 -07:00
Stan Campbell
34193dd142
Merge pull request #22 from rtyler/issues/21-buffer-overrun
...
Avoid double-ocpying and a buffer overrun when pushing a Ruby string
2014-09-04 13:50:33 -07:00
Stan Campbell
3a48bd6f64
Merge pull request #20 from rtyler/issues/11-producer-feedback
...
An initial pass at exposing the async nature of librdkafka upwards
2014-09-04 13:50:14 -07:00
R. Tyler Croy
7085385d7e
Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka
...
This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely
copy the buffer for it's own uses from the Ruby string's C pointer.
This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21
Fixes #21
Fixes #13
2014-09-04 13:03:01 -07:00
R. Tyler Croy
74dfd19399
Remove inadvertantly added changes to msg_delivered function
2014-09-04 11:55:19 -07:00
Stan Campbell
e71ab6cd14
Update the requires for the sample scripts.
2014-09-04 10:52:27 -07:00
R. Tyler Croy
ccc7d20d15
An initial pass at exposign the asynchronous nature of librdkafka up into Ruby
...
This should be enough to get some initial review of the API going. Hooking up a
Hermann::Result class to the Hermann::Producer in Ruby is in this commit but
actually driving the librdkafka reactor underneath is not yet hooked up.
Fixes #11
Fixes #15
Fixes #18
2014-09-03 14:19:09 -07:00
R. Tyler Croy
0eb18479b6
Properly raise an exception when the consumer is given an empty topic
2014-08-30 15:30:11 -07:00
R. Tyler Croy
655cc4463a
Fix almost all of the compiler warnings
...
Some are intentionally left out of the fixing because I believe there's a
failing test case that needs to be added
2014-08-30 15:30:11 -07:00
R. Tyler Croy
f5e0c9384d
Raise an exception instead of attempting to write to an empty topic name
2014-08-30 13:21:33 -07:00
R. Tyler Croy
3effe046f4
Raise an exception instead of exiting process on bad brokers for Herman::Consumer
2014-08-30 13:21:10 -07:00
R. Tyler Croy
2f5e9ae4e3
Make a plethora of whitespace and style consistency fixes to C code
2014-08-30 13:06:19 -07:00
R. Tyler Croy
f3bf216b66
Raise a RuntimeError when a bad partition is given to a consumer
...
Fixes #7
2014-08-30 12:54:53 -07:00
R. Tyler Croy
d8166b7fe4
Raise a runtime error when a bad broker is specified
...
Fixes #6
2014-08-30 12:51:03 -07:00
R. Tyler Croy
3a6a54def6
C code should be formatted with tabs not spaces
2014-08-30 08:40:58 -07:00
R. Tyler Croy
e241f086ee
Restructure the Hermann gem extension build process to use rake-compiler
...
This is a little bit better and cleaner IMO than the hand-crafted rake tasks.
Would you like to know more? <https://github.com/luislavena/rake-compiler >
2014-08-29 18:37:10 -07:00
R. Tyler Croy
2f194b4352
Nuke ruby-version
2014-08-28 21:15:43 -07:00
R. Tyler Croy
ff300e4175
Start refactoring unnecessary logic out of the C-core, starting with Producer
2014-08-28 17:14:38 -07:00
Stan Campbell
ae0b964934
Turn off TRACE and DEBUG by default.
...
JIRA: PART-3886
2014-08-19 13:37:22 -07:00
Stan Campbell
1185d54ff1
Add bulletproofing around the management of config->rk and config->rkt kafka handles.
...
Avoid dereferencing or freeing null handles for Consumer and Producer classes.
JIRA: PART-3886
2014-08-19 13:32:32 -07:00
Stan Campbell
a18272e901
Remove Makefile and mkmf.log as they're generated by extconf.rb and mkmf.
2014-07-25 12:52:39 -07:00
Stan Campbell
80754f6296
Remove bogus binaries from commit.
2014-07-22 11:37:41 -07:00
Stan Campbell
9c04fb1bc1
Add call to rb_thread_blocking_region to release the Ruby GVM lock and allow other threads to execute while we await IO from Kafka.
2014-06-20 10:52:50 -07:00
Stan Campbell
d9c2a93ec8
Add partition and brokers list configuration to the constructors for both Consumer and Producer.
2014-06-16 11:12:44 -07:00
Stan Campbell
aed11dcfac
Update documentation and gem version readying for release.
2014-06-13 10:47:34 -07:00
Stan Campbell
bcccda15a3
Removing more printf's. Removed the UUID to reduce dependencies.
2014-06-06 13:29:38 -07:00
Stan Campbell
874f02e673
Removed some extraneous logging. A little code cleanup.
2014-06-06 12:25:03 -07:00
Stan Campbell
6f0779d1b7
Update producer to use higher performance (don't wait so long in rd_kafka_poll).
2014-06-05 16:22:55 -07:00
Stan Campbell
d7c6387d21
Change the consumer code to use the batched callback method rather than blocking and timing out on single messages.
2014-06-05 16:03:09 -07:00
Stan Campbell
691d591d8c
Code cleanup, added a UUID to the configuration per instance for debugging.
2014-06-05 15:33:08 -07:00
Stan Campbell
a7a35deda9
All state now in configuration associated with instance. However, signal callbacks for trapping exit don't have the proper signature
2014-06-05 09:28:01 -07:00
Stan Campbell
841e33ecb9
Moving rk (instance pointer for the RdKafka producer, consumer, etc.) into per instance config.
2014-06-05 09:11:53 -07:00
Stan Campbell
66c7cbdbea
Added Producer configuration to struct. Simplified prodcfg and conscfg to common.
2014-06-05 09:03:00 -07:00
Stan Campbell
9a9d377863
Continuing edits to move state into the config structs to be held by instances of Producer and Consumer.
2014-06-04 09:26:01 -07:00