69 lines
1.7 KiB
Plaintext
69 lines
1.7 KiB
Plaintext
= filekäfer
|
|
|
|
A little Kafka daemon for shipping logs from the file system to topics.
|
|
|
|
|
|
== Configuration
|
|
|
|
=== JSON
|
|
|
|
This configuration format is compatible with the <<kinesis, Amazon Kinesis Agent>>.
|
|
|
|
.agent.json
|
|
[source,json]
|
|
----
|
|
{
|
|
"checkpointFile": "/tmp/aws-kinesis-agent-checkpoints/main.log",
|
|
"kinesis.endpoint": "https://kinesis.us-west-2.amazonaws.com",
|
|
"flows": [
|
|
{
|
|
"filePattern": "/tmp/aws-kinesis-agent-test1.log*",
|
|
"initialPosition": "END_OF_FILE",
|
|
"kinesisStream": "aws-kinesis-agent-test1",
|
|
"dataProcessingOptions": [
|
|
{
|
|
"optionName": "CSVTOJSON",
|
|
"customFieldNames": [ "field1", "field2", "field3" ],
|
|
"delimiter": "\\t"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"filePattern": "/tmp/aws-kinesis-agent-test2.log*",
|
|
"initialPosition": "START_OF_FILE",
|
|
"kinesisStream": "aws-kinesis-agent-test2",
|
|
"dataProcessingOptions": [
|
|
{
|
|
"optionName": "LOGTOJSON",
|
|
"logFormat": "COMMONAPACHELOG"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
----
|
|
|
|
|
|
Possible file positions are:
|
|
|
|
* `START_OF_FILE`
|
|
* `END_OF_FILE`
|
|
|
|
|
|
== Hacking
|
|
|
|
Working with filekäfer is relatively straightforward, assuming
|
|
link:https://rust-lang.org[Rust] is already present on the system. Docker and
|
|
`docker-compose` are recommended, as is link:https://github.com/edenhill/kafkacat[kafkacat] for debugging.
|
|
|
|
Running `make` should output the available targets with a short description of
|
|
each. For the most part `make run`, `make check`, and `make watch` will be used
|
|
during development of filekäfer.
|
|
|
|
|
|
== Similar tools
|
|
|
|
* link:https://github.com/Qihoo360/logkafka[logkafka]
|
|
[[kinesis]]
|
|
* link:https://github.com/awslabs/amazon-kinesis-agent[amazon-kinesis-agent]
|