rtyler/hotdog
rtyler
/
hotdog
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
156 Commits 1 Branch 16 Tags 292 KiB
Commit Graph

156 Commits

Author SHA1 Message Date
R Tyler Croy c9f24bd17d Make hotdog.connections a gauge 
This feels like a dumb way to accomplish this
 2020-05-09 15:53:57 -07:00
R Tyler Croy 41416d8238 Remove unnecessary connection counter metric 2020-05-09 15:41:13 -07:00
R Tyler Croy b13d529578 Document the error messages 2020-05-09 15:33:09 -07:00
R Tyler Croy b530112b03 Implement some error reporting metrics after sending messages to Kafka 
I'm not entirely pleased with how this code looks right now, it feels dirty but
works 🤷

Fixes #4
 2020-05-09 15:28:32 -07:00
R Tyler Croy 28d454433c Add some metrics for tracking performance with Kafka 
Not yet complete, but checkpointing some work

Fixes #4
 2020-05-09 15:28:29 -07:00
R Tyler Croy f97cfa7a6b Add the table of contents to the README 
It's getting quite long!
 2020-05-08 13:47:57 -07:00
R Tyler Croy 0ad9a9e613
Merge pull request #8 from reiseburo/internal-queueing 
Refactor Kafka connection handling and perform internal queueing
 2020-05-07 08:56:13 -07:00
R Tyler Croy 89bc5d9c53 cargo fmt 2020-05-07 08:52:35 -07:00
R Tyler Croy 7c448c09a8 Significant refactor to better handle kafka messaging 
This commit consolidates the sending of messages to Kafka via a single
"producer" task which is communicated with via crossbeam bounded channels.

hotdog has no reason to create a producer object and connection for every syslog
connection. Structuring the application in this way also allows for log lines to
flow into hotdog with a different throughput rate than being 1:1 tied to what
the Kafka producer can handle.

As it stands right now there are effectively two queues at play here:

* Internal librdkafka sender queue for sending messages asynchronously.
  Basically the number of in-flight messages
* Internal crossbeam channel queue, since the channel we're using is
  intentionally bounded and its size controlled by global.kafka.buffer.

Fixes #5
 2020-05-07 08:44:45 -07:00
R Tyler Croy 1040d50c72 Refactor the passing of settings and metrics through to log handlers 
Instead, each connection will have a ConnectionState struct which should carry
all the needed state the handler requires
 2020-05-07 07:44:47 -07:00
R Tyler Croy 5b4e69366e Add the global.kafka.buffer configuration value 
This value is going to be used for the internal queueing of Kafka messages to
be a little more fault-tolerant
 2020-05-07 07:44:47 -07:00
R Tyler Croy 155c4b144c
Merge pull request #6 from rtyler/rtyler-patch-1 
Add the necessary native libraries for cargo build
 2020-04-22 10:38:54 -07:00
R Tyler Croy 1e60269893
Add the necessary native libraries for cargo build 2020-04-22 10:32:23 -07:00
R Tyler Croy a9e29f40d2
Simple github action for testing 2020-04-20 08:51:14 -07:00
R Tyler Croy 68eebb254d Add an rsyslog configuration for load testing and comparisons 
The rsyslog configuration is intentionally very primitive and doesn't do
anything sensible other than forward a syslog message directly along to the
`logs-unknown` topic. It should be matched with the simple-plaintext from
examples/
 2020-04-19 21:06:40 -07:00
R Tyler Croy 79815ed45b Outline some high-level features for the passers-by 2020-04-19 20:22:50 -07:00
R Tyler Croy 9ed4f29246 Clarify the documentation a bit around the merge and replace actions 2020-04-19 20:09:12 -07:00
R Tyler Croy 8d60d3d827 Add some more documentation on matchers 2020-04-19 20:03:50 -07:00
R Tyler Croy 0cbc0ec48c Update the README with more documentation on the configuration format 2020-04-19 19:41:22 -07:00
R Tyler Croy 95bb84e0d3 Adjust the Kafka configuration to allow raw librdkafka options to pass through 
This should make it much easier to manage underlying configuration or support
changes, since Hotdog will just punt to librdkafka!
 2020-04-19 19:22:21 -07:00
R Tyler Croy 52c9987cd4 Fix a "minor" issue when attempting to forward a message without any prior changes 
Minor in that the message actually doesn't get forwarded, oops
 2020-04-19 19:15:16 -07:00
R Tyler Croy 6051c52510 Starting to add example configurations 2020-04-19 13:34:01 -07:00
R Tyler Croy f4943cadb4 Play with load testing and tsung some more 
This is only for a plaintext server, I've not yet figured out how to properly
load test a TLS enabled server
 2020-04-19 13:32:54 -07:00
R Tyler Croy 4237a51481 Ensure the configuration files can be loaded from a nested path. 
Discovered this bug where using the file stem would result in always using
effectively `./hotdog.yml`
 2020-04-18 21:54:01 -07:00
R Tyler Croy c1e76a0391 Implement basic support for listening for syslog over TLS 
This implementation is still fairly basic. There's more error-handling and
refactoring that I need to do in order to clean this code up, but on its face it
does the basic job of accepting TLS connections without certificate-based
authentication for accepting syslog traffic.

Fixes #1
 2020-04-18 15:32:39 -07:00
R Tyler Croy 84a2931fe9 Add a test TLS certificate and private key and corresponding settings change 
This is the first part of implementing TLS support mentioned in #1.

The expected options in `hotdog.yml` are as follows.

syslog over plaintext TCP:

```
global:
  listen:
    tls:
```

syslog over a TLS encrypted socket

```
global:
  listen:
    tls:
      cert: 'path/to/cert.pem'
      key:  'path/to/key.pem'
```
 2020-04-18 15:01:42 -07:00
R Tyler Croy cfefbe14b2 Add the logs-unknown topic for testing 2020-04-16 20:59:40 -07:00
R Tyler Croy 10f5ff94d4 Add a simple tsung configuration to do some preliminary load testing 2020-04-15 22:21:47 -07:00
R Tyler Croy c7023dad03 Correctly route topics based on variable substitutions for jmespath 
This commit also introduces the functionality where a Forward implies Stop, this
is because we need to `break` out of the action processing during the handling
of the Forward, since `output` is being consumed in the process.

See #4
 2020-04-15 13:04:53 -07:00
R Tyler Croy 19e869c3b4 Properly render and route JSON merges to Kafka 
This also addresses some of the JMESPath routing, as listed in hotdog.yml

Integration tested 👏

Fixes #3, #4
 2020-04-15 11:16:27 -07:00
R Tyler Croy de70b7705b Refactoring and improve the handling of merge actions 
This includes a couple unit tests, but this has not been properly tested in an
integration test environment
 2020-04-14 22:00:36 -07:00
R Tyler Croy 540b1716ee Render variables after merging 
Fixes #3
 2020-04-09 19:03:57 -07:00
R Tyler Croy c7a8ae9e44 Add preliminary support for jmespath matching for rules 
This doesn't yet pull the value out into something that can be used in the merge
or replace actions

Fixes #4
 2020-04-06 21:56:52 -07:00
R Tyler Croy 5d4e30a75b Fix cross-reference link 2020-04-05 20:44:37 -07:00
R Tyler Croy cd6fc4c30a Add a bit more information to the README about the CLI 2020-04-05 17:10:46 -07:00
R Tyler Croy c8c54babbf Add example.log for consistent testing 2020-04-05 17:08:26 -07:00
R Tyler Croy 58fae395d9 Add the `merge` action for merging JSON log lines with pre-defined JSON data 
This does not yet support variable substitution in the JSON, but that's coming
next 🎉
 2020-04-05 17:07:43 -07:00
R Tyler Croy 632ae8b08c Add a simple file testing function to check that the regular expressions match 
This will help developing rules quite a bit!
 2020-04-05 15:19:58 -07:00
R Tyler Croy eb1634b0ff Start processing command line arguments to load the configuration file 2020-04-05 15:05:25 -07:00
R Tyler Croy d466dfc4f9 Incorporate some simple statsd counters for connections and lines received 2020-04-05 14:36:31 -07:00
R Tyler Croy fe06c56c39 Add the "Stop" action to prevent subsequent rules from processing the log 2020-04-05 14:22:13 -07:00
R Tyler Croy fc0b6bf851 Add dipstick so metrics can start to be generated 2020-04-05 14:21:36 -07:00
R Tyler Croy d5bde64c9f Add some debug messages while I'm tinkering here.. 2020-04-05 14:10:28 -07:00
R Tyler Croy b5be3501a8 Adopt serde_regex to automatically deserialize our regex patterns 2020-04-05 14:09:54 -07:00
R Tyler Croy fd99027cba Allow multiple rules to match and modify the incoming data 2020-04-05 13:51:47 -07:00
R Tyler Croy 4f1d1cdeda Iterate through rules and actions, transforming messages as they go 2020-04-04 23:50:36 -07:00
R Tyler Croy e85aab9ffd Add a simple docker-compose.yml for spinning up Kafka/ZooKeeper 2020-04-04 16:18:21 -07:00
R Tyler Croy 6d0a086114 Wire the raw message through for testing purposes to the configured Kafka topic 
This is very very primitive, but hey it works enough for a demo at this point
 2020-04-04 16:17:48 -07:00
R Tyler Croy 6e1d470922 Add a readme and license, just good hygiene there 2020-04-04 14:43:49 -07:00
R Tyler Croy dfc87979be Refactor the settings handling into its own module since it's getting a bit big 2020-04-04 14:42:44 -07:00