Commit Graph

11 Commits

Author SHA1 Message Date
R. Tyler Croy e18ac81468
Introduce codenarc and make the project adhere to some coding standards 2015-09-01 15:50:31 -07:00
R. Tyler Croy d0c99b9a34 Implement the KafkaSpoutTreeWatcher for processing Storm KafkaSpout offset data
Fixes #9
2015-01-28 02:45:49 -08:00
R. Tyler Croy b4b9fe9860 Introduce the AbstractConsumerTreeWatcher to handle watchers on Kafka consumer trees
This gives us two kinds of AbstractTreeWatcher instances, those that watch
special-case subtrees (e.g. the BrokerTreeWatcher) and then those which need to
watch and report Kafka consumer offset information (e.g. StandardTreeWatcher)

References #9
2015-01-28 02:02:59 -08:00
R. Tyler Croy af19abfacb Refactor the handling of TreeCache into the AbstractTreeWatcher itself
This should lay the groundwork for refactoring much of the BrokerTreeWatcher up
into the AbstractTreeWatcher
2015-01-28 01:44:53 -08:00
R. Tyler Croy 5e1a4e11ba Use a ocpy on write list for the consumers map to avoid thrash between threads
Once we've finished caching our consumers map the majority of the operations on
this consumersMap list will be traversals inside of the KafkaPoller. Thus
making the performance hit worth it.

What we don't want to happen is to try to iterate through the consumersmap's
list of consumers while receiving new ZK childEvents

Fixes #6
2015-01-26 06:52:54 -08:00
R. Tyler Croy 49c2d7fc1d Add logback as a dependency and gut the printlns from the codebase 2015-01-26 04:11:27 -08:00
R. Tyler Croy 293ebb6fa9 Prefer callback lists instead of a single callback per object
This doesn't really matter but I prefer this approach from a style vantage
point
2015-01-19 13:15:00 -08:00
R. Tyler Croy 8b792d72fd Properly use the passed in AbstractMap for the StandardTreeWatcher 2015-01-19 13:08:01 -08:00
R. Tyler Croy c53413d5b9 Introduce TopicPartition for acting as a Hash key for (topic, partition_id) tuples
This means the consumersMap that we're going to keep track of will have the key
of a TopicPartition, e.g.: ConcurrentHashMap<TopicPartition, List<ConsumerOffset>>

When we receive the data from the Kafka meta-data calls (to be added) all we'll
need to do is create the right TopicPartition and walk the list of
ConsumerOffset instances to start reporting metrics
2015-01-19 10:12:41 -08:00
R. Tyler Croy c4bd4ef9c2 Push a HashMap through into the TreeWatchres to start keeping tabs on consumer offsets
This data structure isn't fully bake dyet, but at least gives some organization
to the data we're caching out of the Zookeeper tree
2015-01-18 10:25:04 -08:00
R. Tyler Croy afd2ee8567 Introduce the TreeWatcher objects for transforming events into something tangible
References #2
2015-01-17 16:51:26 -08:00