1.7 KiB
1.7 KiB
<html lang="en">
<head>
</head>
</html>
filekäfer
A little Kafka daemon for shipping logs from the file system to topics.
Configuration
JSON
This configuration format is compatible with the Amazon Kinesis Agent.
agent.json
{
"checkpointFile": "/tmp/aws-kinesis-agent-checkpoints/main.log",
"kinesis.endpoint": "https://kinesis.us-west-2.amazonaws.com",
"flows": [
{
"filePattern": "/tmp/aws-kinesis-agent-test1.log*",
"initialPosition": "END_OF_FILE",
"kinesisStream": "aws-kinesis-agent-test1",
"dataProcessingOptions": [
{
"optionName": "CSVTOJSON",
"customFieldNames": [ "field1", "field2", "field3" ],
"delimiter": "\\t"
}
]
},
{
"filePattern": "/tmp/aws-kinesis-agent-test2.log*",
"initialPosition": "START_OF_FILE",
"kinesisStream": "aws-kinesis-agent-test2",
"dataProcessingOptions": [
{
"optionName": "LOGTOJSON",
"logFormat": "COMMONAPACHELOG"
}
]
}
]
}Possible file positions are:
-
START_OF_FILE -
END_OF_FILE
Hacking
Working with filekäfer is relatively straightforward, assuming
Rust is already present on the system. Docker and
docker-compose are recommended, as is kafkacat for debugging.
Running make should output the available targets with a short description of
each. For the most part make run, make check, and make watch will be used
during development of filekäfer.