rtyler
/
filekaefer
1
0
Fork 0
Code Issues Pull Requests Releases Activity
A little Kafka daemon for shipping logs from the file system to topics.
15 Commits 1 Branch 0 Tags 72 KiB
Rust 88.1%
Makefile 5.8%
AsciiDoc 4.2%
SQL 1.9%
Go to file
R Tyler Croy 3e0924820d
Commit some work in progress thoughts around the checkpoint and follower module 		2019-11-18 19:00:44 -08:00
migrations Add the checkpoints schema for diesel 2019-11-16 15:24:45 -08:00
src Commit some work in progress thoughts around the checkpoint and follower module 2019-11-18 19:00:44 -08:00
.gitignore Correct the Cargo build to exclude libmysqlclient 2019-11-16 15:55:01 -08:00
Cargo.lock Correct the Cargo build to exclude libmysqlclient 2019-11-16 15:55:01 -08:00
Cargo.toml Correct the Cargo build to exclude libmysqlclient 2019-11-16 15:55:01 -08:00
HACKING.adoc Set up a local sqlite database for tracking checkpoints 2019-11-16 15:17:46 -08:00
LICENSE.txt Set up a local sqlite database for tracking checkpoints 2019-11-16 15:17:46 -08:00
Makefile Add some details on how to hack on filekäfer to the readme for posterity 2019-04-14 15:22:26 -07:00
README.adoc Add some details on how to hack on filekäfer to the readme for posterity 2019-04-14 15:22:26 -07:00
diesel.toml Set up a local sqlite database for tracking checkpoints 2019-11-16 15:17:46 -08:00
docker-compose.yml Add docker-compose.yml for local testing/development 2019-04-14 15:19:25 -07:00

README.adoc

<html lang="en"> <head> </head>

filekäfer

A little Kafka daemon for shipping logs from the file system to topics.

Configuration

JSON

This configuration format is compatible with the Amazon Kinesis Agent.

agent.json
{
  "checkpointFile": "/tmp/aws-kinesis-agent-checkpoints/main.log",
  "kinesis.endpoint": "https://kinesis.us-west-2.amazonaws.com",
  "flows": [
    {
      "filePattern": "/tmp/aws-kinesis-agent-test1.log*",
      "initialPosition": "END_OF_FILE",
      "kinesisStream": "aws-kinesis-agent-test1",
      "dataProcessingOptions": [
        {
          "optionName": "CSVTOJSON",
          "customFieldNames": [ "field1", "field2", "field3" ],
          "delimiter": "\\t"
        }
      ]
    },
    {
      "filePattern": "/tmp/aws-kinesis-agent-test2.log*",
      "initialPosition": "START_OF_FILE",
      "kinesisStream": "aws-kinesis-agent-test2",
      "dataProcessingOptions": [
        {
          "optionName": "LOGTOJSON",
          "logFormat": "COMMONAPACHELOG"
        }
      ]
    }
  ]
}

Possible file positions are:

  • START_OF_FILE

  • END_OF_FILE

Hacking

Working with filekäfer is relatively straightforward, assuming Rust is already present on the system. Docker and docker-compose are recommended, as is kafkacat for debugging.

Running make should output the available targets with a short description of each. For the most part make run, make check, and make watch will be used during development of filekäfer.

Last updated 2024-04-24 05:02:15 UTC
</html>