reiseburo
/
verspaetung
mirror of https://github.com/reiseburo/verspaetung
1
0
Fork 0
Code Issues Releases Wiki Activity
97 Commits 2 Branches 9 Tags 355 KiB
Commit Graph

53 Commits

Author SHA1 Message Date
Christian Meier 7a6ef71eef refactor the BrokerTreeWatcher 
split things into smaller pieces and as preparation of having only
ONE TreeWatcher running which itself dispatch the events to the interresting
parties.

Sponsored by Lookout Inc.
 2015-11-10 22:42:07 +01:00
Christian Meier 618aaf915e make the connection or disconnection of consumers failsafe 
i.e. ignore errors on disconnect and do not register consumers which
fail to connect
 2015-11-10 22:15:57 +01:00
Christian Meier 66c0c332bb make is proper POJO and remove json magic to allow easier testing 2015-11-10 22:15:57 +01:00
Christian Meier 4c655995f9 nicer logging 2015-11-10 22:15:57 +01:00
Christian Meier dc918a700e not needed 2015-11-10 22:15:57 +01:00
R. Tyler Croy fb4f152fed
Bump the minor version for release, updating the group too 
Renaming the entire source tree now that this is under reiseburo
 2015-11-07 12:39:25 -08:00
Christian Meier 98af950498 be nice and close all resources on shutdown 2015-11-03 20:40:49 +01:00
Christian Meier 439e8f635e declare private methods as such 
helps to understand what methods could be used by other threads
 2015-11-03 20:40:49 +01:00
Christian Meier 45d98cb5f6 allow new nodes to be added when they pop up 
also ensure the broker list is synchronized in both the KafkaPoller
as well the BrokerTreeWatcher threads
 2015-11-03 20:40:48 +01:00
Christian Meier 59c8e79f4e reduce polling speed in case kafka is responding with exceptions 
just double the interval on each successive error and reset it to 1 sec
once a request succeeded again. maximum polling inteval is about half
an hour.
 2015-11-03 20:40:41 +01:00
R. Tyler Croy 3778a24044
Add the --exclude CLI option for filtering out unwanted consumer groups 
Fixes #34
 2015-09-01 17:10:43 -07:00
R. Tyler Croy 7b918d448a
Don't use groovy strings where we don't need to 2015-09-01 16:29:34 -07:00
R. Tyler Croy e18ac81468
Introduce codenarc and make the project adhere to some coding standards 2015-09-01 15:50:31 -07:00
R. Tyler Croy fc97f67100
Add support for a customized delay in seconds between reports (defaulting to 5s) 2015-09-01 14:41:16 -07:00
R. Tyler Croy b6bbcf92a4
Don't reconnect if we have invalid data in our brokerConsumerMap, just warn about it 
Fixes #30
 2015-08-14 15:11:16 -07:00
R. Tyler Croy be33333bfa
Bail out and try to force a reconnect if our broker's consumer is nulled 
Still need to explore how we're getting into this state, but this is a start to
handling that scenario more gracefully

References #130
 2015-08-14 10:50:45 -07:00
R. Tyler Croy 5df6083e6d
Log a single error line for KafkaException exceptions when polling 
Fixes #31
 2015-08-14 10:13:17 -07:00
R. Tyler Croy e01e699861
Move the hacking docs out and link to groovydocs properly 2015-07-04 15:08:22 -07:00
R. Tyler Croy f968a2c1c8
Move the README over to asciidoc and wire together publishing asciidoc to gh-pages 2015-07-04 15:02:35 -07:00
R. Tyler Croy f053d9b68c Catch and log exceptions coming from the dumpMetadata() call 
Fixes #23
 2015-03-22 11:08:40 -07:00
R. Tyler Croy 3a9caa2535 Avoid computing negative values for offsets, making zero the lowest possible value 
Fixes #25
 2015-03-22 10:56:26 -07:00
R. Tyler Croy dc33298435 Groovy's getStrings() method with string interpolation doesn't do what you think it does 
groovy:000> user = "Ron"
    ===> Ron
    groovy:000> "hello ${user}"
    ===> hello Ron
    groovy:000> "hello ${user}".getStrings()
    ===> [hello , ]
    groovy:000> "hello ${user}".toString()
    ===> hello Ron

Fixes #24
 2015-03-20 15:13:40 -07:00
R. Tyler Croy 38e8c62e00 Properly report metrics to datadog with the appropriate tags 
This requires a much more recent version of our metrics-datadog library but
does result in the right values being reported into datadog.
 2015-03-20 10:58:45 -07:00
R. Tyler Croy 71347594ca Major refactor to support dropwizard-metrics as the means of outputting metrics 
This commit changes the structure of Verspaetung pretty dramatically to allow
for the registering of Gauges for the various offsets

With this change the KafkaPoller is pushing the latest offsets into a map and the
ZK consumer tree watchers are pushing consumer offsets into a separate map.

References #17
 2015-02-06 08:34:19 -08:00
R. Tyler Croy 7b13050ee2 Restructure KafkaPoller to make catching and logging exceptions easier 
I've not cleaned this up enough to make things easier to test, partially
because mocking the Kafka client APIs is a giant pain in the ass

Fixes #16
 2015-01-30 02:45:16 -08:00
R. Tyler Croy 0fa983d0e8 Add support for prefixing the metrics with a CLI supplied option 
Fixes #15
 2015-01-30 02:45:16 -08:00
R. Tyler Croy 95c244867a Leave a skipped test in place for later 
Going to get back to #14
 2015-01-30 02:45:16 -08:00
R. Tyler Croy 8995351bba Parse the consumer group from the KafkaSpout ZK path instead of using the JSON 
The JSON in the Znode is actually the name of the topology, not necessarily the
name of the consumer group used by a KafkSpout

Fixes #9
 2015-01-28 03:05:30 -08:00
R. Tyler Croy d0c99b9a34 Implement the KafkaSpoutTreeWatcher for processing Storm KafkaSpout offset data 
Fixes #9
 2015-01-28 02:45:49 -08:00
R. Tyler Croy b4b9fe9860 Introduce the AbstractConsumerTreeWatcher to handle watchers on Kafka consumer trees 
This gives us two kinds of AbstractTreeWatcher instances, those that watch
special-case subtrees (e.g. the BrokerTreeWatcher) and then those which need to
watch and report Kafka consumer offset information (e.g. StandardTreeWatcher)

References #9
 2015-01-28 02:02:59 -08:00
R. Tyler Croy af19abfacb Refactor the handling of TreeCache into the AbstractTreeWatcher itself 
This should lay the groundwork for refactoring much of the BrokerTreeWatcher up
into the AbstractTreeWatcher
 2015-01-28 01:44:53 -08:00
R. Tyler Croy 0706895af1 Refactor some of the KafkaPoller setup out and add support for --dry-run 
Fixes #10
 2015-01-28 01:13:13 -08:00
R. Tyler Croy a444f49b46 Invoke the onDelta callbacks for zero deltas too 
It turns out that datadog doesn't indicate that the delta is shrinking unless
you report a zero value.
 2015-01-26 09:40:16 -08:00
R. Tyler Croy 18e747788e Add a simple heartbeat to the configured statsd host for every second9 
Fixes #4
 2015-01-26 07:00:07 -08:00
R. Tyler Croy 5e1a4e11ba Use a ocpy on write list for the consumers map to avoid thrash between threads 
Once we've finished caching our consumers map the majority of the operations on
this consumersMap list will be traversals inside of the KafkaPoller. Thus
making the performance hit worth it.

What we don't want to happen is to try to iterate through the consumersmap's
list of consumers while receiving new ZK childEvents

Fixes #6
 2015-01-26 06:52:54 -08:00
R. Tyler Croy 4f16c3dfd1 Avoid NPEs when starting up and scanning brokers 
The problem was we were trying to parse the JSON of /brokers/ids' when the
treecache found that node.

Fixes #8
 2015-01-26 06:20:46 -08:00
R. Tyler Croy a022252157 Add some simple command line overrides for ZK and statsd hosts 
Fixes #5
 2015-01-26 06:15:35 -08:00
R. Tyler Croy c15c25534c Remove unnecessary synchronized statement 2015-01-26 04:17:28 -08:00
R. Tyler Croy 49c2d7fc1d Add logback as a dependency and gut the printlns from the codebase 2015-01-26 04:11:27 -08:00
R. Tyler Croy 6d82735b3c Ignore "owners" and other subtrees from the Kafka High Level Consumer ZK space 
References #2
 2015-01-26 04:06:27 -08:00
R. Tyler Croy ba2fa562ac Stupidly and blindly fire a bunch of crap into statsd on localhost 
This is not the final product by a long shot, just needed to figure out how
things would look inside datadog
 2015-01-20 13:23:01 -08:00
R. Tyler Croy 5fe4d8efaa Hack together the actual delta reporting, UNCLEAN CODE ALERT 
Still very much in the experimental phase, need to refactor KafkaPoller a lot
at this point and decide which code should live where and give it plenty of
tests
 2015-01-19 16:22:22 -08:00
R. Tyler Croy 193b147064 Exploratory testing, a veritable boatload of garbage and hacks 
This will at least create some stupid consumers and try to fetch the latest
offsets for a bunch of stupid topics.

The Kafka/Scala internals are so immensely frustrating
 2015-01-19 15:55:47 -08:00
R. Tyler Croy 293ebb6fa9 Prefer callback lists instead of a single callback per object 
This doesn't really matter but I prefer this approach from a style vantage
point
 2015-01-19 13:15:00 -08:00
R. Tyler Croy 8b792d72fd Properly use the passed in AbstractMap for the StandardTreeWatcher 2015-01-19 13:08:01 -08:00
R. Tyler Croy 7e593f1235 Introducing the BrokerTreeWatcher to track changes to the broker list in Zookeeper 
This commit includes a lot of work in progress kind of code. STill
experimenting with how to bind the events from the Zookeeper event-driven
system, in with the KafkaPoller busy-wait-loop system.
 2015-01-19 11:45:22 -08:00
R. Tyler Croy c53413d5b9 Introduce TopicPartition for acting as a Hash key for (topic, partition_id) tuples 
This means the consumersMap that we're going to keep track of will have the key
of a TopicPartition, e.g.: ConcurrentHashMap<TopicPartition, List<ConsumerOffset>>

When we receive the data from the Kafka meta-data calls (to be added) all we'll
need to do is create the right TopicPartition and walk the list of
ConsumerOffset instances to start reporting metrics
 2015-01-19 10:12:41 -08:00
R. Tyler Croy c4bd4ef9c2 Push a HashMap through into the TreeWatchres to start keeping tabs on consumer offsets 
This data structure isn't fully bake dyet, but at least gives some organization
to the data we're caching out of the Zookeeper tree
 2015-01-18 10:25:04 -08:00
R. Tyler Croy afd2ee8567 Introduce the TreeWatcher objects for transforming events into something tangible 
References #2
 2015-01-17 16:51:26 -08:00
R. Tyler Croy 86878d5b10 Nuke some old classes that didn't make any sense 2015-01-17 15:54:51 -08:00