2015-09-02 18:52:08 +00:00
|
|
|
= Beetle
|
|
|
|
|
|
|
|
|
|
Beetle is a somewhat higher level Java API on top of the client libraries
|
|
|
|
|
distributed distributed with link:http://kafka.apache.org[Apache Kafka]. The
|
|
|
|
|
goal of this library is not to replace the use of those libraries, but to wrap
|
|
|
|
|
the library in a more easy to use package.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
== System Requirements
|
|
|
|
|
|
|
|
|
|
* JDK7 or later
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
== Hacking
|
|
|
|
|
|
|
|
|
|
This project uses link:http://gradle.org[Gradle] so building and testing should
|
|
|
|
|
be as easy as executing:
|
|
|
|
|
|
|
|
|
|
% ./gradlew
|
|
|
|
|
|
2015-09-03 14:32:24 +00:00
|
|
|
|
|
|
|
|
== Design/Notes/Thoughts
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
NOTE: Right now this section is very much just a brain-dump/work-in-progress
|
|
|
|
|
|
|
|
|
|
What is fundamentally missing from the upstream Kafka clients is an evented
|
|
|
|
|
view on the world. Despite Zookeeper and Kafka's models effectively being
|
|
|
|
|
event-driven,
|
|
|
|
|
link:https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example[implementing
|
|
|
|
|
a lower-level SimpleConsumer] utilizes busy-loops and rather disjointed logic
|
|
|
|
|
for reconnects and error handling. The link:https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example[higher-level
|
|
|
|
|
consumer API] is also awkward to use as far as receiving messages (using an
|
|
|
|
|
iterator) and handling parallel operations (stuffing a thread pool somewhere
|
|
|
|
|
for receiving).
|
|
|
|
|
|
|
|
|
|
A high-level Kafka consumer API maps rather nicely to the
|
|
|
|
|
link:https://github.com/ReactiveX/RxJava[RxJava] usage model of Observers and
|
|
|
|
|
Subscribers, e.g.g
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.Consumer.java
|
|
|
|
|
[source, java]
|
|
|
|
|
----
|
|
|
|
|
/* Using Java 8 Lambda syntax for conciseness' sake */
|
2015-09-03 20:21:19 +00:00
|
|
|
LocateBrokers.zookeeper("localhost:2181")
|
2015-09-03 14:32:24 +00:00
|
|
|
.map(broker -> TopicSubscription("some-topic"))
|
2015-09-03 20:21:19 +00:00
|
|
|
/* assuming a custom consume() operator exists in Beetle */
|
|
|
|
|
.consume(message -> doSomethingWithMessage(message))
|
2015-09-03 14:32:24 +00:00
|
|
|
.map(message -> message.commitOffset());
|
|
|
|
|
----
|
2015-09-03 21:22:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
== Similar Projects
|
|
|
|
|
|
|
|
|
|
. link:https://github.com/cjdev/kafka-rx[kafka-rx]: Scala-based client which
|
|
|
|
|
provides a push alternative to kafka's pull-based stream
|
