Commit Graph

72 Commits

Author SHA1 Message Date
R. Tyler Croy 246f27e76d
Add some code coverage while we're here 2016-02-03 05:32:57 -08:00
jakesandlund 6946d4d82c Fix java_producer_spec for modified push_single signature 2015-09-22 18:36:10 +00:00
jakesandlund c29bb5e4d0 Move to_java into java_producer to make push_single signature match MRI 2015-09-16 15:54:42 +00:00
cory and jakesandlund e8703e1df4 Add support for passing partition_key in MRI 2015-09-14 16:11:43 +00:00
Ben Osheroff 9edc4b9301 move hermann_lib -> hermann_rdkafka / Hermann::Provider::RDKafka 2015-06-23 09:01:31 -07:00
jway 2aecf369e6 fix integration tests 2015-06-01 09:42:37 -07:00
Thomas 7100c84059 Fix TypeError conversion of Array into String
When running rake spec:integration with /fixtures/integration.yml

Hermann::Lib::Producer #connect should connect
     Failure/Error: subject(:producer) { Hermann::Lib::Producer.new(brokers) }
     TypeError: no implicit conversion of Array into String
     # ./spec/hermann_lib/producer_spec.rb:10:in `initialize'
2015-05-19 11:00:25 +02:00
R. Tyler Croy 7b4ea2f2df Properly handle pulling in fixtures code across Ruby versions 2015-05-07 13:56:36 -07:00
R. Tyler Croy 3fda5ec3f3 Rework the dependencies and some of the code to support retro rubies better 2015-05-07 13:18:41 -07:00
Ben Osheroff 3fbe6969fb update tests with InvalidOffsetError 2015-04-28 15:14:14 -07:00
Ben Osheroff 752123b83e allow choosing of an inital offset in the MRI consumer 2015-04-28 10:17:09 -07:00
Ben Osheroff 21190d7acd randomize integration test topic 2015-04-28 10:09:30 -07:00
Ben Osheroff 3792ee206b unify jruby/mri Consumer#initialize
While we do require different options between jruby & mri, there is a
common "topic" option.  The rest we pass in the options hash.
2015-04-27 09:32:16 -07:00
Lee Butterman f059643efc Add support for a partition key for producers.
Fixes lookout/Hermann#88
2015-04-06 17:35:38 -07:00
jway 96c0a74bf5 0.20.4 add large file support, fix bug passing options 2015-04-04 11:47:29 -07:00
Dirk Koehler 163c9d70c6 Cleanup jar dependencies
- move Jarfile dependency into gemspec
- upgrade curator to 2.7.1
- fix tests
- remove transient dependencies
2015-03-26 22:58:13 -07:00
R. Tyler Croy aeaec0269c Merge pull request #76 from rtyler/issues/75-native-zookeeper
Implement a Java-native Zookeeper interface with Curator
2014-11-21 21:01:24 -08:00
R. Tyler Croy 495ac2687a Refactor out the block support from #each_brokers and bring back `timeout` for the ZkGemImpl
[1] pry(main)> d = Hermann::Discovery::Zookeeper.new('zk0.REDACTED.com:2181')
    => #<Hermann::Discovery::Zookeeper:0x57186526
     @impl=
      #<Hermann::Discovery::Zookeeper::CuratorImpl:0x391d1e33
       @curator=
        #<Java::OrgApacheCuratorFrameworkImps::CuratorFrameworkImpl:0x5287ba5f>>,
     @zookeepers="zk0.REDACTED.com:2181">
    [2] pry(main)> d.get_brokers
    log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.CuratorFrameworkImpl).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    => ["kafka0.REDACTED.com:6667",
     "kafka1.REDACTED.com:6667"]
    [3] pry(main)>

    [1] pry(main)> d = Hermann::Discovery::Zookeeper.new('zk0.REDACTED.com:2181')
    => #<Hermann::Discovery::Zookeeper:0x7b6141f0
     @impl=
      #<Hermann::Discovery::Zookeeper::ZkGemImpl:0x4fb64a52
       @zookeepers="zk0.REDACTED.com:2181">,
     @zookeepers="zk0.REDACTED.com:2181">
    [2] pry(main)> d.get_brokers
    log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    => ["kafka0.REDACTED.com:6667",
     "kafka1.REDACTED.com:6667"]
    [3] pry(main)>
2014-11-21 20:39:11 -08:00
R. Tyler Croy 136e0ace47 Add suppot for optionally using Curator if it's on the classpath for broker discovery
This change makes the loading of the 'zk' gem optional and implicit, as well as
the reliance on Curator being available in the JRuby runtime's classpath.

If neither Zookeeper client implementation are available, an exception will be
raised when attmepting to use Zookeeper-based discovery

Fixes #75
2014-11-21 18:06:56 -08:00
jway 6300705af9 fixup exception handling in consumer 2014-11-13 16:23:40 -08:00
R. Tyler Croy 0f7714175b Correct test failures with the API changes to Hermann exceptions and some underlying expectations 2014-10-31 12:02:17 -07:00
R. Tyler Croy 61c0426cbe Catch all Java related exceptions and raise them up out of Hermann::Producer#push_single 2014-10-31 11:24:11 -07:00
R. Tyler Croy 228358edc1 Add a failing integration test for pumping protobufs through Hermann 2014-10-30 10:46:18 -07:00
R. Tyler Croy 7f4de34be0 Refactor the Hermann::Producer Java-based integration test to respect integration.yml
I've also refactored/cleaned it up a bit to make sure it runs properly on my machine
2014-10-30 10:46:18 -07:00
R. Tyler Croy 3c9c234c43 Update the Hermann::Discovery::Zookeeper API to return an Array of brokers more suitable for Hermann::Producer 2014-10-30 10:05:34 -07:00
jway 132dbb94a9 pass string to JavaProducer 2014-10-29 16:59:34 -07:00
R. Tyler Croy 4ea21d5add Account for a subtle internal API change between the C and Java Producers 2014-10-28 18:18:05 -07:00
jway 69202c21b2 allow integration test to run only in jruby 2014-10-28 11:14:14 -07:00
jway b48698035b update integration example with zookeepers 2014-10-28 08:35:33 -07:00
jway 9acce0a64b add shutdown
* producer/consumer integration tests
2014-10-27 21:16:17 -07:00
jway 5e0079ce05 allow passing topic into consume method 2014-10-25 09:12:47 -07:00
R. Tyler Croy 8c01040f5d Wrap calls to the underlying Java producer with some Ruby-based exception handling
Fixes #58
2014-10-24 11:47:51 -07:00
jway 972fe47410 add simple java consumer 2014-10-24 10:07:27 -07:00
R. Tyler Croy e0070b942b Properly reap children in tests on MRI and Java
Fixes #54
2014-10-15 12:21:29 -07:00
jway 63c64f8ea3 add reaping of java promises 2014-10-15 11:32:51 -07:00
jway 9ca7340b21 allow user to pass in opts 2014-10-14 15:17:54 -07:00
R. Tyler Croy 249f2739a7 Properly propagate the optionally passed in topic on #push when recursing through Arrays
This commit also moves tests back out of the platform specific scope and makes
the internal #push_single API more consistent to alllow easier testing.

Fixes #49
2014-10-14 10:54:52 -07:00
R. Tyler Croy 5b6fa5075f Support passing in a topic for every #push in MRI
This commit also fixes a memory leak with the JRuby version of the gem where we
were not properly cleaning up children objects and holding onto references of
them forever

Fixes #46
2014-10-14 09:51:24 -07:00
jway a6a1148221 allow java producer push to change topics 2014-10-13 15:17:10 -07:00
jway cd16650ad7 refactor zookeeper lookup into Discovery class 2014-10-10 11:12:22 -07:00
jway 354fbde7fa add zookeeper broker lookup to Java provider 2014-10-08 15:25:32 -07:00
R. Tyler Croy 4cfb92b697 Clean up the execution of RSpecs to run the platform-specific tests in the right place 2014-10-08 09:45:07 -07:00
jway 3151b3ef4b push method returns executing Promise 2014-10-07 15:27:36 -07:00
jway 17b6db78b1 updates specs, fixup producer exceptional handling 2014-10-06 11:23:32 -07:00
jway 07a1dcec8e add specs for java_producer 2014-10-06 08:08:30 -07:00
jway b898709042 first pass adding java kafka provider 2014-10-03 12:38:38 -07:00
R. Tyler Croy 7af7e3ec5b Avoid call to rb_string_value_cstr() which doesn't handle binary strings from Ruby properly
Fixes #32
2014-09-10 20:00:07 -07:00
R. Tyler Croy 254efad00a Add a failing integration test for binary data
Just a quick reproduction case for #32
2014-09-10 11:04:14 -07:00
R. Tyler Croy 341e176b2f Propagate system broker errors into Hermann::Result objects
Provided that Producer#tick_reactor has been invoked, if there are underlying
errors like "1/1 Brokers down" then this code will ensure that outstanding
Herman::Result children of Herman::Producer will have their error status updated.

Fixes #25
2014-09-10 11:04:14 -07:00
R. Tyler Croy 218c035793 Expose an #errored? method which gets set if the general error callback is invoked
Fixes #25
2014-09-09 17:37:05 -07:00