17b6db78b1
updates specs, fixup producer exceptional handling
2014-10-06 11:23:32 -07:00
de03fa4ebc
add require jars to top level module
2014-10-06 09:33:35 -07:00
07a1dcec8e
add specs for java_producer
2014-10-06 08:08:30 -07:00
b898709042
first pass adding java kafka provider
2014-10-03 12:38:38 -07:00
f1ef2772e2
Merge pull request #38 from rtyler/issues/8-bundle-librdkafka
...
Bundle librdkafka 0.8.4 with the gem
2014-09-12 16:32:43 -07:00
7bf9cf745d
Disable rubinius builds in travis
2014-09-12 16:26:37 -07:00
44efb9e2b5
Roll back to using 0.8.4 APIs since we're bundling now
2014-09-12 16:15:04 -07:00
5de9612882
Bundle a known good version of librdkafka (0.8.4) with Hermann
...
This reduces the need for the librdkafka system dependency and results in
statically linking librdkafka.a against the C extension.
There's a few hacks in extconf.rb specifically to deal with librdkafka's build
system, as well as apply checksumming to the downloaded file
Fixes #8
2014-09-12 16:13:19 -07:00
afd14dec80
Revert back to using rd_kafka_conf_set_dr_cb() which is 0.8.3 compatible
...
Fixes #36
2014-09-12 14:24:53 -07:00
d8f72fc9f2
Bump to 0.16.0 for the next release
2014-09-12 14:12:53 -07:00
e7f0f900ad
Merge pull request #35 from rtyler/issues/32-binary-data
...
Avoid call to rb_string_value_cstr() which doesn't handle binary strings properly
2014-09-11 11:34:24 -07:00
7af7e3ec5b
Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly
...
Fixes #32
2014-09-10 20:00:07 -07:00
838904a035
Merge pull request #33 from rtyler/issues/25-producer-errors
...
Propagate systemic errors from librdkafka up into Herman::Result objects
2014-09-10 12:48:07 -07:00
254efad00a
Add a failing integration test for binary data
...
Just a quick reproduction case for #32
2014-09-10 11:04:14 -07:00
341e176b2f
Propagate system broker errors into Hermann::Result objects
...
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.
Fixes #25
2014-09-10 11:04:14 -07:00
cea9161bbb
Merge pull request #31 from rtyler/issues/25-producer-errors
...
Add lots more machinery to help identify errors outside of a `push` command
2014-09-10 10:26:50 -07:00
218c035793
Expose an #errored? method which gets set if the general error callback is invoked
...
Fixes #25
2014-09-09 17:37:05 -07:00
288653845a
Introduce a wrapper struct to make message delivery callbacks able to change Producer data
...
This will help with determining whether we're no longer connected due to
failures after a push of a message
2014-09-09 16:45:27 -07:00
a0697f5b66
Refactor out the necessity for Hermann::Timeout and just rely in a Ruby-based busyloop
2014-09-09 16:45:17 -07:00
9014551d10
Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value
...
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly
Fixes #25
2014-09-09 14:40:10 -07:00
4d7e9f3b4e
Add Hermann::Timeout abstraction for 1.8/1.9 compatible timeout support
2014-09-09 14:40:10 -07:00
0f9681ab91
Hold onto the Hermann::Lib::Producer pointer inside of the rd_kafka_conf_t
2014-09-09 14:40:10 -07:00
acec652d8b
Switch out all these ifdef TRACE statements for a cleaner TRACER() macro
2014-09-09 14:40:10 -07:00
08f47503ed
Add a Rake task for running integration tests
2014-09-09 11:18:08 -07:00
128ec66db0
Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls
...
This also introduces some more/better error handling in Hermann::Lib::Producer#tick
Fixes #24
2014-09-09 11:11:07 -07:00
08d1aab811
Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests
2014-09-09 10:51:49 -07:00
c2dfcf30f2
Add support for running integration tests through RSpec against a pre-configured broker/topic
...
This will make it a bit easier to run local tests in a more automated fashion
against some non-public broker hostnames
2014-09-09 10:47:22 -07:00
99eb52e781
Add pry to the gemfile for better local testing/dev
2014-09-09 10:45:55 -07:00
46d62a26cb
Remove exit(..) calls and replace by raising a runtime exception to Ruby.
2014-09-09 09:19:28 -07:00
77248706e8
Use RSTRING_LEN instead of _LENINT which isn't present on 1.8.7
...
Fixes #30
2014-09-09 09:09:50 -07:00
5cf9f352e1
Merge pull request #23 from rtyler/issues/11-producer-feedback
...
Tie Hermann::Result to the librdkafka reactor
2014-09-04 15:09:44 -07:00
9bfd7ad2e6
Tie Hermann::Result#value to the underlying reactor to bring values up to Ruby
...
This ensures that we're getting async values out of librdkafka into the calling
Ruby thread. Currently errors aren't being brought up properly, but we're getting there
Example:
[14:47:30] tyler:Hermann git:(issues/11-producer-feedback*) $ pry -I lib -r 'hermann/producer'
[1] pry(main)> p = Hermann::Producer.new('topic', 'kafka0.REDACTED.com:6667')
=> #<Hermann::Producer:0x00000803b3b450
@brokers="kafka0.REDACTED.com:6667",
@children=[],
@internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
@topic="topic">
[2] pry(main)> r = p.push('hello world!')
=> #<Hermann::Result:0x00000803b8cb20
@producer=
#<Hermann::Producer:0x00000803b3b450
@brokers="kafka0.REDACTED.com:6667",
@children=[#<Hermann::Result:0x00000803b8cb20 ...>],
@internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
@topic="topic">,
@reason=nil,
@state=:unfulfilled,
@value=nil>
[3] pry(main)> r.state
=> :unfulfilled
[4] pry(main)> r.value
ticking rdkafka reactor
ticked
=> "hello world!"
[5] pry(main)> r.state
=> :fulfilled
[6] pry(main)> r.rejected?
=> false
[7] pry(main)>
[14:47:56] tyler:Hermann git:(issues/11-producer-feedback*) $
Fixes #11
2014-09-04 14:51:41 -07:00
ed36300265
Pass a pointer to a Hermann::Result into the message delivery callback
...
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.
What's still missing is producer level error calbacks
Fixes #11
Fixes #15
Fixes #17
2014-09-04 14:31:29 -07:00
3a3df235a7
Properly pass the block to Consumer#consume into the C layer
2014-09-04 14:29:54 -07:00
dc9a057eaa
Add a simple script to validate big message behavior
...
References #21
2014-09-04 14:29:29 -07:00
34193dd142
Merge pull request #22 from rtyler/issues/21-buffer-overrun
...
Avoid double-ocpying and a buffer overrun when pushing a Ruby string
2014-09-04 13:50:33 -07:00
3a48bd6f64
Merge pull request #20 from rtyler/issues/11-producer-feedback
...
An initial pass at exposing the async nature of librdkafka upwards
2014-09-04 13:50:14 -07:00
7085385d7e
Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka
...
This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely
copy the buffer for it's own uses from the Ruby string's C pointer.
This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21
Fixes #21
Fixes #13
2014-09-04 13:03:01 -07:00
74dfd19399
Remove inadvertantly added changes to msg_delivered function
2014-09-04 11:55:19 -07:00
e71ab6cd14
Update the requires for the sample scripts.
2014-09-04 10:52:27 -07:00
8aff9226b7
Add example scripts testing producer and consumer functionality.
2014-09-04 10:27:47 -07:00
ccc7d20d15
An initial pass at exposign the asynchronous nature of librdkafka up into Ruby
...
This should be enough to get some initial review of the API going. Hooking up a
Hermann::Result class to the Hermann::Producer in Ruby is in this commit but
actually driving the librdkafka reactor underneath is not yet hooked up.
Fixes #11
Fixes #15
Fixes #18
2014-09-03 14:19:09 -07:00
3081848bde
Add the gitter chat link to the readme
2014-09-02 14:01:55 -07:00
9477cd10d3
Merge pull request #4 from rtyler/bro-do-you-even
...
Add some basic sanity checks for Hermann gem
2014-09-02 13:35:31 -07:00
0eb18479b6
Properly raise an exception when the consumer is given an empty topic
2014-08-30 15:30:11 -07:00
655cc4463a
Fix almost all of the compiler warnings
...
Some are intentionally left out of the fixing because I believe there's a
failing test case that needs to be added
2014-08-30 15:30:11 -07:00
f5e0c9384d
Raise an exception instead of attempting to write to an empty topic name
2014-08-30 13:21:33 -07:00
3effe046f4
Raise an exception instead of exiting process on bad brokers for Herman::Consumer
2014-08-30 13:21:10 -07:00
2f5e9ae4e3
Make a plethora of whitespace and style consistency fixes to C code
2014-08-30 13:06:19 -07:00
f3bf216b66
Raise a RuntimeError when a bad partition is given to a consumer
...
Fixes #7
2014-08-30 12:54:53 -07:00
jway
Stan Campbell
R. Tyler Croy
Stan Campbell