Stupid evening research project

This commit is contained in:
R Tyler Croy 2019-04-04 16:22:15 -07:00
parent dc21b9c6dc
commit 51f99860b0
No known key found for this signature in database
GPG Key ID: E5C92681BEF6CEA2
2 changed files with 91 additions and 0 deletions

View File

@ -0,0 +1,85 @@
layout: post
title: "Publishing to Azure Event Hubs from Rust"
- rust
- kafka
- azure
Turn back now, this blog post is so niche that it's statistically impossible
for you to find this useful. Last night I was thinking about building a little
app which needed to deal with an event stream, and started poking around the
Azure Event Hubs documentation. I noticed that they _apparently_ can now [speak
which means I can use my existing Kafka library tooling, nice! Since I was
already working with Kafka and Rust for another little project, I took a quick
detour and tried to see if I could publish to an Event Hub over Kafka, from
Rust. As luck would have it, I can!
The one quirk with Azure Event Hubs compared to how most people use Kafka was
the SASL/TLS authentication and encryption, was a bit tricky to use with the
[rdkafka]( crate, which built on top of the
fantastic [librdkafka]( SASL/TLS is
seems to have become the defacto standard for running Kafka-as-a-Service like
AWS MSK or Confluent Cloud, so it is wonderful to figure out how to use the
rdkafka crate in this environment.
The relevant code from [my
gist]( which
demonstrates this functionality is the `ClientConfig` for the producer:
* NOTE: `connection` is the full connection string for the Event Hub
let producer: FutureProducer = ClientConfig::new()
.set("bootstrap.servers", &brokers)
.set("", "true")
.set("", "5000")
.set("security.protocol", "sasl_ssl")
* The following setting -must- be set for librdkafka, but doesn't seem to do anything
* worthwhile,. Since we're not relying on kerberos, let's just give it some junk :)
.set("sasl.kerberos.kinit.cmd", "echo 'who wants kerberos?!'")
* Another unused setting which is required to be present
.set("sasl.kerberos.keytab", "keytab")
.set("sasl.mechanisms", "PLAIN")
* The username that Azure Event Hubs uses for Kafka is really this
.set("sasl.username", "$ConnectionString")
* NOTE: Depending on your system, you may need to change this to adifferent location
.set("", "/etc/ssl/ca-bundle.pem")
.set("sasl.password", &connection)
.expect("Producer creation error");
With those settings in place, any of the existing Rust rdkafka exampels should
work just fine, though I haven't tested the consumer API just yet.
I should also note that the Azure Event Hubs "Standard" tier supports enabling
the Kafka protocol, but their "Basic" tier does not, which is too bad. The
"Basic" tier is cheaply priced enough to make it reasonable for a
hobby-project's event stream.
I'm looking forward to playing a bit more with Rust and Kafka in the future for
a myriad of reasons. The most compelling of which are "serverless"
environments, such as that provided by the [AWS Lambda Rust
runtime]( The pricing of
AWS Lambda encourages fast (pricing per 100ms) and small (pricing per MB of
memory) makes it an ideal candidate for Rust.
Whether you're considering Azure Event Hubs, or any other SASL/TLS-based Kafka
interface, now you know how to use it from Rust!

tag/ Normal file
View File

@ -0,0 +1,6 @@
layout: tag_page
title: "Tag: rust"
tag: rust
robots: noindex