reiseburo
/
hermann
mirror of https://github.com/reiseburo/hermann
1
0
Fork 0
Code Issues Releases Wiki Activity
238 Commits 9 Branches 20 Tags 604 KiB
Commit Graph

238 Commits

Author SHA1 Message Date
R. Tyler Croy 218c035793 Expose an #errored? method which gets set if the general error callback is invoked 
Fixes #25
 2014-09-09 17:37:05 -07:00
R. Tyler Croy 288653845a Introduce a wrapper struct to make message delivery callbacks able to change Producer data 
This will help with determining whether we're no longer connected due to
failures after a push of a message
 2014-09-09 16:45:27 -07:00
R. Tyler Croy a0697f5b66 Refactor out the necessity for Hermann::Timeout and just rely in a Ruby-based busyloop 2014-09-09 16:45:17 -07:00
R. Tyler Croy 9014551d10 Make use of Hermann::Timeout when ticking the Producers reactor to safely block for a value 
This adds a requirement for 1.8.7 clients to install the SystemTimer gem for
timing out on Hermann::Result#value(timeout) to block and timeout correctly

Fixes #25
 2014-09-09 14:40:10 -07:00
R. Tyler Croy 4d7e9f3b4e Add Hermann::Timeout abstraction for 1.8/1.9 compatible timeout support 2014-09-09 14:40:10 -07:00
R. Tyler Croy 0f9681ab91 Hold onto the Hermann::Lib::Producer pointer inside of the rd_kafka_conf_t 2014-09-09 14:40:10 -07:00
R. Tyler Croy acec652d8b Switch out all these ifdef TRACE statements for a cleaner TRACER() macro 2014-09-09 14:40:10 -07:00
R. Tyler Croy 08f47503ed Add a Rake task for running integration tests 2014-09-09 11:18:08 -07:00
R. Tyler Croy 128ec66db0 Propagate the timeout from Hermann::Producer#tick_reactor down into the rd_kafka_poll calls 
This also introduces some more/better error handling in Hermann::Lib::Producer#tick

Fixes #24
 2014-09-09 11:11:07 -07:00
R. Tyler Croy 08d1aab811 Fix a test-discovered issue when calling Hermann::Lib::Producer#tick without having started requests 2014-09-09 10:51:49 -07:00
R. Tyler Croy c2dfcf30f2 Add support for running integration tests through RSpec against a pre-configured broker/topic 
This will make it a bit easier to run local tests in a more automated fashion
against some non-public broker hostnames
 2014-09-09 10:47:22 -07:00
R. Tyler Croy 99eb52e781 Add pry to the gemfile for better local testing/dev 2014-09-09 10:45:55 -07:00
Stan Campbell 46d62a26cb Remove exit(..) calls and replace by raising a runtime exception to Ruby. 2014-09-09 09:19:28 -07:00
R. Tyler Croy 77248706e8 Use RSTRING_LEN instead of _LENINT which isn't present on 1.8.7 
Fixes #30
 2014-09-09 09:09:50 -07:00
Stan Campbell 5cf9f352e1 Merge pull request #23 from rtyler/issues/11-producer-feedback 
Tie Hermann::Result to the librdkafka reactor
 2014-09-04 15:09:44 -07:00
R. Tyler Croy 9bfd7ad2e6 Tie Hermann::Result#value to the underlying reactor to bring values up to Ruby 
This ensures that we're getting async values out of librdkafka into the calling
Ruby thread. Currently errors aren't being brought up properly, but we're getting there

Example:

    [14:47:30] tyler:Hermann git:(issues/11-producer-feedback*) $ pry -I lib -r 'hermann/producer'
    [1] pry(main)> p = Hermann::Producer.new('topic', 'kafka0.REDACTED.com:6667')
    => #<Hermann::Producer:0x00000803b3b450
    @brokers="kafka0.REDACTED.com:6667",
    @children=[],
    @internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
    @topic="topic">
    [2] pry(main)> r = p.push('hello world!')
    => #<Hermann::Result:0x00000803b8cb20
    @producer=
    #<Hermann::Producer:0x00000803b3b450
    @brokers="kafka0.REDACTED.com:6667",
    @children=[#<Hermann::Result:0x00000803b8cb20 ...>],
    @internal=#<Hermann::Lib::Producer:0x00000803b3b3d8>,
    @topic="topic">,
    @reason=nil,
    @state=:unfulfilled,
    @value=nil>
    [3] pry(main)> r.state
    => :unfulfilled
    [4] pry(main)> r.value
    ticking rdkafka reactor
    ticked
    => "hello world!"
    [5] pry(main)> r.state
    => :fulfilled
    [6] pry(main)> r.rejected?
    => false
    [7] pry(main)>
    [14:47:56] tyler:Hermann git:(issues/11-producer-feedback*) $

Fixes #11
 2014-09-04 14:51:41 -07:00
R. Tyler Croy ed36300265 Pass a pointer to a Hermann::Result into the message delivery callback 
Provided that /something/ is ticking the librdkafka reactor underneath, the
Hermann::Result will be updated with the message delivery status when that's
completed.

What's still missing is producer level error calbacks

Fixes #11
Fixes #15
Fixes #17
 2014-09-04 14:31:29 -07:00
R. Tyler Croy 3a3df235a7 Properly pass the block to Consumer#consume into the C layer 2014-09-04 14:29:54 -07:00
R. Tyler Croy dc9a057eaa Add a simple script to validate big message behavior 
References #21
 2014-09-04 14:29:29 -07:00
Stan Campbell 34193dd142 Merge pull request #22 from rtyler/issues/21-buffer-overrun 
Avoid double-ocpying and a buffer overrun when pushing a Ruby string
 2014-09-04 13:50:33 -07:00
Stan Campbell 3a48bd6f64 Merge pull request #20 from rtyler/issues/11-producer-feedback 
An initial pass at exposing the async nature of librdkafka upwards
 2014-09-04 13:50:14 -07:00
R. Tyler Croy 7085385d7e Avoid double-ocpying and a buffer overrun when pushing a Ruby string into Kafka 
This approach to shuffling the message buffer into librdkafka relies on librdkafka to safely
copy the buffer for it's own uses from the Ruby string's C pointer.

This has been tested with buffers 2x the size of the SIGSEGV mentioned in #21

Fixes #21
Fixes #13
 2014-09-04 13:03:01 -07:00
R. Tyler Croy 74dfd19399 Remove inadvertantly added changes to msg_delivered function 2014-09-04 11:55:19 -07:00
Stan Campbell e71ab6cd14 Update the requires for the sample scripts. 2014-09-04 10:52:27 -07:00
Stan Campbell 8aff9226b7 Add example scripts testing producer and consumer functionality. 2014-09-04 10:27:47 -07:00
R. Tyler Croy ccc7d20d15 An initial pass at exposign the asynchronous nature of librdkafka up into Ruby 
This should be enough to get some initial review of the API going. Hooking up a
Hermann::Result class to the Hermann::Producer in Ruby is in this commit but
actually driving the librdkafka reactor underneath is not yet hooked up.

Fixes #11
Fixes #15
Fixes #18
 2014-09-03 14:19:09 -07:00
R. Tyler Croy 3081848bde Add the gitter chat link to the readme 2014-09-02 14:01:55 -07:00
Stan Campbell 9477cd10d3 Merge pull request #4 from rtyler/bro-do-you-even 
Add some basic sanity checks for Hermann gem
 2014-09-02 13:35:31 -07:00
R. Tyler Croy 0eb18479b6 Properly raise an exception when the consumer is given an empty topic 2014-08-30 15:30:11 -07:00
R. Tyler Croy 655cc4463a Fix almost all of the compiler warnings 
Some are intentionally left out of the fixing because I believe there's a
failing test case that needs to be added
 2014-08-30 15:30:11 -07:00
R. Tyler Croy f5e0c9384d Raise an exception instead of attempting to write to an empty topic name 2014-08-30 13:21:33 -07:00
R. Tyler Croy 3effe046f4 Raise an exception instead of exiting process on bad brokers for Herman::Consumer 2014-08-30 13:21:10 -07:00
R. Tyler Croy 2f5e9ae4e3 Make a plethora of whitespace and style consistency fixes to C code 2014-08-30 13:06:19 -07:00
R. Tyler Croy f3bf216b66 Raise a RuntimeError when a bad partition is given to a consumer 
Fixes #7
 2014-08-30 12:54:53 -07:00
R. Tyler Croy d8166b7fe4 Raise a runtime error when a bad broker is specified 
Fixes #6
 2014-08-30 12:51:03 -07:00
R. Tyler Croy 3a6a54def6 C code should be formatted with tabs not spaces 2014-08-30 08:40:58 -07:00
R. Tyler Croy e8a9ebe7a4 Depend on build instead of just compile in the default task 2014-08-29 18:37:41 -07:00
R. Tyler Croy e241f086ee Restructure the Hermann gem extension build process to use rake-compiler 
This is a little bit better and cleaner IMO than the hand-crafted rake tasks.

Would you like to know more? <https://github.com/luislavena/rake-compiler>
 2014-08-29 18:37:10 -07:00
R. Tyler Croy 9a24317c7f Major rework of the gemspec 
This change:

 - removes the platform specific binding (Ruby will run extconf.rb on platforms)
 - adds myself to the authors
 - Pulls version from the gem itself
 - includes all the files in lib that are added to the git tree
 2014-08-29 13:52:36 -07:00
R. Tyler Croy 3efa2c481b Avoid packaging the gem with a spurious object file 2014-08-29 13:47:49 -07:00
R. Tyler Croy 47914ad5e5 Add a simple Rake default task to build and test 2014-08-29 11:00:35 -07:00
R. Tyler Croy 4ca84f4ca9 Comment out tests which exit the Ruby interpreter until they can be fixed 2014-08-29 08:41:57 -07:00
R. Tyler Croy 603aba612a Implement enough of Herman::Consumer#consume to reproduce an exit(1) 2014-08-29 08:41:13 -07:00
R. Tyler Croy 941adaef3b Add test case that exits the ruby interpreter (to be fixed real soon) 2014-08-28 21:18:27 -07:00
R. Tyler Croy 164d87132b Default to documentation format 2014-08-28 21:17:44 -07:00
R. Tyler Croy 2f194b4352 Nuke ruby-version 2014-08-28 21:15:43 -07:00
R. Tyler Croy ff300e4175 Start refactoring unnecessary logic out of the C-core, starting with Producer 2014-08-28 17:14:38 -07:00
R. Tyler Croy 5e34b82498 Remove unused empty test case 2014-08-28 16:21:00 -07:00
R. Tyler Croy 96ff104061 Add some basic test cases to make sure that at least the library is getting loaded 2014-08-28 16:20:46 -07:00
R. Tyler Croy fbdf9b6055 Add rspec to the project for testing basic Hermann functionality 2014-08-28 16:13:12 -07:00