Hotdog is a syslog-to-Kafka forwarder which aims to get log entries into Apache Kafka as quickly as possible.

Go to file

R Tyler Croy bd3eb822e5 Update the workflow to deal with pull requests to main		2020-11-04 13:38:15 -08:00
.cargo	Add the `simd` feature to enable simd-json	2020-09-07 09:50:35 -07:00
.github/workflows	Update the workflow to deal with pull requests to main	2020-11-04 13:38:15 -08:00
contrib	Properly load either RSA or PKCS8 formatted keys	2020-05-09 19:03:14 -07:00
examples/simple-plaintext	Adjust the Kafka configuration to allow raw librdkafka options to pass through	2020-04-19 19:22:21 -07:00
scripts	Allow the generate script to take arbitrary user input	2020-06-22 11:06:01 -07:00
src	Change the prototype of from_str to allow &mut str in order to allow for simd-json	2020-09-07 09:50:35 -07:00
test/configs	Fix minor configuration issue I noticed while testing. `tls` should be optional	2020-05-29 16:52:59 -07:00
.gitignore	…
Cargo.lock	Upgrade a number of dependencies	2020-10-29 10:06:06 -07:00
Cargo.toml	Upgrade a number of dependencies	2020-10-29 10:06:06 -07:00
LICENSE.txt	…
README.adoc	Tidy up the doc	2020-05-31 22:13:14 -07:00
docker-compose.yml	Add the logs-unknown topic for testing	2020-04-16 20:59:40 -07:00
example.log	Correctly route topics based on variable substitutions for jmespath	2020-04-15 13:04:53 -07:00
hotdog-loadtest.yml	Add the loadtest profile	2020-06-22 11:51:26 -07:00
hotdog.yml	Add the optional status configuration for health checks and other info	2020-05-31 15:13:18 -07:00
tsung.xml	Play with load testing and tsung some more	2020-04-19 13:32:54 -07:00

README.adoc

🌭 Hotdog!

Hotdog is a syslog-to-Kafka forwarder which aims to get log entries into Apache Kafka as quickly as possible.

It listens for syslog messages over plaintext or TLS-encrupted TCP connection and depending on the defined Rules it will route and even modify messages on their way into a configured Kafka broker.

Table of Contents

Features
Installation
Configuration
- Global
- Rules
Metrics
Development
- Profiling
Similar Projects

Features

syslog over plaintext or TLS-encrypted TCP connections.
Rules and Actions for matching, modifying, and routing syslog messages based on the message content.
Rich integration with Kafka with full configuration passthrough for librdkafka
Built-in metrics integration for daemon health reporting.

Hotdog 0.1.0
R Tyler Croy <rtyler+hotdog@brokenco.de
Forward syslog over to Kafka with ease

USAGE:
    hotdog [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --config <FILE>       Sets a custom config file [default: hotdog.yml]
    -t, --test <TEST_FILE>    Test a log file against the configured rules

Installation

Hotdog can be installed by grabbing a released binary. The system which will run hotdog must have libsasl2 installed, e.g.:

Ubuntu

sudo apt-get install libsasl2-2

openSUSE

sudo zypper install libsasl2-3

Configuration

Hotdog is configured by the hotdog.yml file, which has a very fluid syntax at the moment. The two main sections are the global and rules blocks.

Rules defined in the configuration can be tested against an example log file in order to verify that the right rules are matching the expected log inputs, for example:

❯ RUST_LOG=info ./target/debug/hotdog -t example.log
Line 1 matches on:
         - Regex: ^hello\s+(?P<name>\w+)?
         - Regex: .*
Line 2 matches on:
         - Regex: .*
Line 3 matches on:
         - Regex: .*
Line 4 matches on:
         - JMESPath: meta.topic
         - Regex: .*

Global

The global configuration configures hotdog itself. The listen, kafka, and metrics keys are all required by default in order for hotdog to start properly.

Listen

The global.listen configuration is required and will determine on which address and port hotdog will listen. The tls configuration key is required to function as well. When tls is left blank, hotdog will listen for syslog messages in plaintext on the specified port.

hotdog.yml

global:
  listen:
    address: '127.0.0.1'
    port: 1514
    tls:

TLS

The global.listen.tls configuration section can be used to enable syslog-over-TLS support from hotdog. Currently the only two valid keys for this section are cert and key, both of which should be absolute or relative paths to PEM-encoded files on disk.

Certificate and Key files can be created with certtool --generate-privkey --outfile ca-key.pem

hotdog.yml

global:
  listen:
    tls:
      cert: './a/path.crt'
      key: './a/path.key'
      # ca is optional and when provided will ensure certificate validation
      # happens
      ca: './a/ca.crt'

Status

The global.status is an optional configuration entry which will enable the launching of an HTTP status server on the specified addresss and port.

JSON formatted statistics can be retrieved on /stats.

hotdog.yml

global:
  status:
    address: '127.0.0.1'
    port: 8585

Kafka

A global.kafka configuration is required in order for hotdog to function properly. The two main configuration values are conf and topic.

hotdog.yml

global:
  kafka:
    conf:
      bootstrap.servers: 'localhost:9092'
      client.id: 'hotdog'
    topic: 'logs'

Buffer

Default: 1024

global.kafka.buffer may contain a number indicating the size of the internal queue for sending messages to Kafka. This queue represents the number of internal messages hotdog will buffer during Kafka availability issues.

This value is not the same as the librdkafka queue.buffering.max.messages configuration, which governs the number of in-flight messages which can be sent at any given time to the Kafka broker(s). To set that variable, include it in the Conf section documented below.

Caution

If the internal Kafka queue has been filled up, new log lines received by hotdog will be discarded.

Conf

global.kafka.conf should contain a map of librdkafka configuration values. hotdog will expect every key and value to be a String. These configuration values are passed right on to the underlying librdkafka client connection, so whatever librdkafka supports, hotdog supports!

timeout_ms

Default: 30_000

global.kafka.timeout_ms is an optional configuration which defines the timeout in milliseconds for hotdog to make an initial connection to the configured Kafka brokers.

Topic

global.kafka.topic may contain a string value which is to be considered the "default topic" for the Forward action.

Metrics

The global.metrics configuration tells hotdog where to send its own internal metrics The only currently supported metrics format is statsd.

If your environment doesn’t use statsd or you do not wish to report metrics, set the statsd value to an invalid host and port.

hotdog.yml

global:
  metrics:
    statsd: 'localhost:8125'

Status

The global.status configuration is fully optional but when it is enabled hotdog will spin up an HTTP server on the configured address and port in order to provide real-time status information about the daemon’s runtime to HTTP clients.

hotdog.yml

global:
  status:
    address: 'localhost'
    port: 8585

Rules

Hotdog’s rules define how it should handle and route the syslog messages it receives. In the hotdog.yml, the rules must be defined as an array of maps.

Each rule is expected to a "matcher" (either regex or jmespath), the field upon which the matcher should apply, and the actions defining how the message should be handled.

hotdog.yml

rules:
  - jmespath: 'meta.topic'
    field: msg
    actions:
      - type: forward
        topic: '{{value}}'

  # Catch-all, send everything else to a "logs-unknown" topic
  - regex: '.*'
    field: msg
    actions:
      - type: forward
        topic: 'logs-unknown'

Table 1. Supported Fields
Name	Notes
`msg`	The actual message sent along from the syslog server
`hostname`	The sender’s hostname, if available.
`appname`	The logging application, if available, which created the syslog entry
`facility`	The syslog logging facility, if available, which was used to create the syslog message. For example `kern`, `user`, `auth`, etc.
`severity`	The severity of the syslog message, if available. For example: `notice`, `err`, `crit`, etc.

Matching with regular expressions

The regex matcher instructs hotdog to match the field against the defined regular expression, which must follow the syntax of the regex crate.

The matcher supports named groups in the regular expression, which are then exposed to actions such as merge and replace.

Caution

Named groups will override any built-in variables at the time of substitution, so be careful you are not naming your groups anything which might overlap with the built-in variable names

Matching with JMESPath

hotdog also supports matching on JSON based messages with JMESPath via the jmespath matcher. In order for a match, the log message must be a valid JSON object or array. The value of the match is also then exposed as a variable named value, which can be used in actions such as merge or replace.

Variables

Some actions, such as Replace, can perform variable substitutions on log line. The variables available are a combination of the built-in variables listed below, and whatever named groups exist in the regex field of the Rules.

Table 2. Built-in Variables
Name	Description
`msg`	The original log line message sent along from the syslog sender.
`version`	The version of `hotdog` which is processing the message.
`iso8601`	The ISO-8601 timestamp of when the message was processed.

Actions

Actions determine what hotdog should do with the given log line when it receives it.

Forward

The forward action implies the Stop action when used, since the internally tracked output buffer is flushed when it is sent to Kafka.

Merge

The merge action will only work when the log line is a JSON object. JSON arrays, or other arbitrary strings will not merge properly, and cause all subsequent actions for the given rule to be aborted.

Table 3. Parameters
Key	Value
`json`	A YAML map which will be merged with the JSON object deserialized from the matched log line.

hotdog.yml

    actions:
      - type: merge
        json:
          meta:
            hotdog:
              version: '{{version}}'
              timestamp: '{{iso8601}}'

Replace

The template may utilize the matched and built-in variables in order to generate a modified message. The output is only available to subsequent actions defined after the replace action. Subsequent rules in the chain will not utilize this generated message.

Table 4. Parameters
Key	Value
`template`	A Handlebars-style template which can be used to output a modified message.

hotdog.yml

  - regex: '^hello\s+(?P<name>\w+)?'
    actions:
      - type: replace
        template: |
          Why hello there {{name}}!

Stop

The stop action does nothing more than stop processing on the message. It is not particularly useful except in cases where hotdog should match on a message and then effectively discard it.

Metrics

hotdog is designed to emit Statsd metrics to the statsd endpoint configured in the Metrics section. Each metric will be prefixed under hotdog.*.

Key	Description
`hotdog.connections`	Gauge tracking the number of connections
`hotdog.lines`	Counter tracking the number of lines received by `hotdog`
`hotdog.kafka.submitted`	Counter tracking the number of messages submitted to Kafka
`hotdog.kafka.submitted.<topicname>`	Counter tracking the number of messages submitted to each Kafka topic
`hotdog.kafka.producer.sent`	Timer which tracks the amount of time it takes to actually write messages to Kafka
`hotdog.kafka.producer.error.*`	Counters which count the number of different errors encountered while sending messages to Kafka. The types of possible metric names depends on the RDKafkaError enumeration from the underlying library.
`hotdog.error.log_parse`	Number of the log lines received which could not be parsed as RFCC 5424 syslog lines.
`hotdog.error.full_internal_queue`	Count tracking the number of log lines which were dropped due to a full internal queue, Typically indicates an issue between `hotdog` and the Kafka brokers.
`hotdog.error.internal_push_failed`	Number of lines dropped because the could not be sent into the internal queue.
`hotdog.error.topic_parse_failed`	Number of lines dropped because the configured dynamic topic could not be parsed properly (typically indicates a configuration error).
`hotdog.error.merge_of_invalid_json`	Count of lines which could not have a merge action applied as configured due to a configuration error
`hotdog.error.merge_target_not_json`	Count of lines received for a merge action which were not JSON, and therefore could not be merged.

Development

Hotdog is tested against the latest Rust stable. A simple cargo build should compile a working hotdog binary for your platform.

On Linux systems it is easy to test with:

logger --server 127.0.0.1  -T -P 1514 "hello world"
logger --server 127.0.0.1  -T -P 1514 -f example.log

For TLS connections, you can use the openssl s_client command:

echo  '<13>1 2020-04-18T15:16:09.956153-07:00 coconut tyler - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="505061"] hello world' | openssl s_client -connect localhost:6514

Profiling

Profiling hotdog is best done on a Linux host with the perf tool, e.g.

RUST_LOG=info perf record --call-graph dwarf -- ./target/debug/hotdog -c ./hotdog.yml
perf report -ng

Similar Projects

hotdog was originally motivated by challenges with rsyslog, a desire for a simple configuration, and the need for built-in metrics.

Some other similar projects which can be used to get logs into Kafka:

logstash
syslog-ng
vector
syslogger, which doesn’t process messages itself, but rather integrates with rsyslog.

Last updated 2024-04-27 01:43:54 UTC

</html>

README.adoc Unescape Escape

🌭 Hotdog!

Features

Installation

Configuration

Global

Listen

TLS

Status

Kafka

Buffer

Conf

timeout_ms

Topic

Metrics

Status

Rules

Matching with regular expressions

Matching with JMESPath

Variables

Actions

Forward

Merge

Replace

Stop

Metrics

Development

Profiling

Similar Projects

README.adoc