The Apache Kafka C/C++ library

Go to file

Magnus Edenhill fdb69ce598 Added RD_KAFKA_OFFSET_TAIL(cnt) to start consuming 'cnt' messages from end. An application can now easily consume the last 'cnt' messages from a partition.		2014-07-30 23:32:20 +02:00
examples	Added RD_KAFKA_OFFSET_TAIL(cnt) to start consuming 'cnt' messages from end.	2014-07-30 23:32:20 +02:00
mklove	mklove update: fixes compiles on osx 10.9 (issue #49 )	2014-06-14 23:43:20 +02:00
rpm	Added RPM package spec file for building RPM package	2014-03-17 14:40:57 +07:00
src	Added RD_KAFKA_OFFSET_TAIL(cnt) to start consuming 'cnt' messages from end.	2014-07-30 23:32:20 +02:00
src-cpp	Update rdkafkacpp_int.h	2014-06-25 13:41:29 +08:00
tests	Added "unique" test ids for message validation	2014-07-30 23:32:20 +02:00
.dir-locals.el	…
.gitignore	Provide linker script for c++ library aswell	2014-03-21 13:56:07 +07:00
.travis.yml	Updated travis-ci to use configure	2014-03-16 14:55:24 +07:00
CONFIGURATION.md	Added produce.offset.report to propagate message offset back to producer application	2014-06-16 14:54:05 +02:00
INTRODUCTION.md	C++ info update	2014-03-26 08:52:54 +07:00
LICENSE	…
LICENSE.pycrc	…
LICENSE.snappy	Added Andi Kleen's Snappy compression C implementation	2013-09-05 23:44:23 +02:00
Makefile	Integrated mklove configure	2014-03-16 14:38:19 +07:00
README.md	Added link to Hermann	2014-06-17 00:37:57 +02:00
configure	mklove update	2014-03-21 13:58:52 +07:00
configure.librdkafka	linker-script: check for perl, else disable linker script	2014-05-05 22:00:47 +02:00
lds-gen.pl	Now builds on Solaris (issue #110 )	2014-05-04 23:38:52 +02:00

README.md

librdkafka - Apache Kafka C/C++ client library

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 800000 msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

NOTE: The master branch is actively developed, use latest release for production use.

Apache Kafka 0.8 support:

Branch: master
Producer: supported
Consumer: supported
Compression: snappy and gzip
Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
ZooKeeper: not supported
C API: Stable, ABI safe, not backwards compatible with 0.7
C++ API: Testing
Tests: Regression tests in tests/ directory.
Statistics: JSON formatted, see rd_kafka_conf_set_stats_cb in rdkafka.h.
Status: Stable

Apache Kafka 0.7 support:

Branch: 0.7
Producer: supported
Consumer: supported
Compression: not supported
ZooKeeper: not supported
C API: backwards compatible with 0.6
Status: Stable

Apache Kafka 0.6 support:

Branch: 0.6
Producer: supported
Consumer: supported
Compression: not supported
ZooKeeper: not supported
Status: Testing

#Users of librdkafka#

Wikimedia's varnishkafka - Varnish cache web log producer
kafkacat - Apache Kafka swiss army knife
redBorder
Headweb
Produban's log2kafka - Web log producer
phpkafka - PHP
node-kafka - Node.js
node-kafkacat - Node.js
haskakafka - Haskell
haskell-kafka - Haskell
Hermann - Ruby
large unnamed financial institution
Let me know if you are using librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib

Instructions

Building

  ./configure
  make
  sudo make install

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The C API is documented in src/rdkafka.h

The C++ API is documented in src-cpp/rdkafkacpp.h

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services