e18ac81468
Introduce codenarc and make the project adhere to some coding standards
2015-09-01 15:50:31 -07:00
fc97f67100
Add support for a customized delay in seconds between reports (defaulting to 5s)
2015-09-01 14:41:16 -07:00
38e8c62e00
Properly report metrics to datadog with the appropriate tags
...
This requires a much more recent version of our metrics-datadog library but
does result in the right values being reported into datadog.
2015-03-20 10:58:45 -07:00
71347594ca
Major refactor to support dropwizard-metrics as the means of outputting metrics
...
This commit changes the structure of Verspaetung pretty dramatically to allow
for the registering of Gauges for the various offsets
With this change the KafkaPoller is pushing the latest offsets into a map and the
ZK consumer tree watchers are pushing consumer offsets into a separate map.
References #17
2015-02-06 08:34:19 -08:00
0fa983d0e8
Add support for prefixing the metrics with a CLI supplied option
...
Fixes #15
2015-01-30 02:45:16 -08:00
d0c99b9a34
Implement the KafkaSpoutTreeWatcher for processing Storm KafkaSpout offset data
...
Fixes #9
2015-01-28 02:45:49 -08:00
af19abfacb
Refactor the handling of TreeCache into the AbstractTreeWatcher itself
...
This should lay the groundwork for refactoring much of the BrokerTreeWatcher up
into the AbstractTreeWatcher
2015-01-28 01:44:53 -08:00
0706895af1
Refactor some of the KafkaPoller setup out and add support for --dry-run
...
Fixes #10
2015-01-28 01:13:13 -08:00
18e747788e
Add a simple heartbeat to the configured statsd host for every second9
...
Fixes #4
2015-01-26 07:00:07 -08:00
a022252157
Add some simple command line overrides for ZK and statsd hosts
...
Fixes #5
2015-01-26 06:15:35 -08:00
49c2d7fc1d
Add logback as a dependency and gut the printlns from the codebase
2015-01-26 04:11:27 -08:00
ba2fa562ac
Stupidly and blindly fire a bunch of crap into statsd on localhost
...
This is not the final product by a long shot, just needed to figure out how
things would look inside datadog
2015-01-20 13:23:01 -08:00
5fe4d8efaa
Hack together the actual delta reporting, UNCLEAN CODE ALERT
...
Still very much in the experimental phase, need to refactor KafkaPoller a lot
at this point and decide which code should live where and give it plenty of
tests
2015-01-19 16:22:22 -08:00
193b147064
Exploratory testing, a veritable boatload of garbage and hacks
...
This will at least create some stupid consumers and try to fetch the latest
offsets for a bunch of stupid topics.
The Kafka/Scala internals are so immensely frustrating
2015-01-19 15:55:47 -08:00
293ebb6fa9
Prefer callback lists instead of a single callback per object
...
This doesn't really matter but I prefer this approach from a style vantage
point
2015-01-19 13:15:00 -08:00
8b792d72fd
Properly use the passed in AbstractMap for the StandardTreeWatcher
2015-01-19 13:08:01 -08:00
7e593f1235
Introducing the BrokerTreeWatcher to track changes to the broker list in Zookeeper
...
This commit includes a lot of work in progress kind of code. STill
experimenting with how to bind the events from the Zookeeper event-driven
system, in with the KafkaPoller busy-wait-loop system.
2015-01-19 11:45:22 -08:00
c53413d5b9
Introduce TopicPartition for acting as a Hash key for (topic, partition_id) tuples
...
This means the consumersMap that we're going to keep track of will have the key
of a TopicPartition, e.g.: ConcurrentHashMap<TopicPartition, List<ConsumerOffset>>
When we receive the data from the Kafka meta-data calls (to be added) all we'll
need to do is create the right TopicPartition and walk the list of
ConsumerOffset instances to start reporting metrics
2015-01-19 10:12:41 -08:00
c4bd4ef9c2
Push a HashMap through into the TreeWatchres to start keeping tabs on consumer offsets
...
This data structure isn't fully bake dyet, but at least gives some organization
to the data we're caching out of the Zookeeper tree
2015-01-18 10:25:04 -08:00
afd2ee8567
Introduce the TreeWatcher objects for transforming events into something tangible
...
References #2
2015-01-17 16:51:26 -08:00
70f0d61e7b
Exploring using curator's TreeCache for grabbing an entire ZK sub-tree
2015-01-17 14:42:24 -08:00
2e8cac342f
Start exploring walking the ZTree with Curator
...
This isn't anything worth a damned, just naively walking a tree in Zookeeper
which is absurdly slow
2015-01-09 18:17:45 -08:00
18c99ad79c
Add the simple Gradle infrastructure necessary to build and test the project
2015-01-09 15:55:35 -08:00
R. Tyler Croy