rust
/
rust-rdkafka
mirror of https://github.com/fede1024/rust-rdkafka
1
0
Fork 0
Code Issues Releases Wiki Activity
990 Commits 45 Branches 43 Tags 16 MiB
Commit Graph

990 Commits

Author SHA1 Message Date
David Blewett e69c2aa40c
Merge pull request #587 from Magister/fix/native_config_get 
Fix panic on getting config value from NativeConfig
 2024-01-19 09:59:15 -05:00
David Blewett eacf17389e
Merge pull request #639 from Swatinem/to-string 
Use `CStr::to_string_lossy` in Base{Consumer,Producer}
 2024-01-19 09:58:34 -05:00
David Blewett bb76b5bd40 Release v0.36.2. 2024-01-16 17:00:49 -05:00
David Blewett ed293167f7
Merge pull request #646 from fede1024/davidblewett/kafka-36 
Add Kafka 3.6 to the integration test matrix.
 2024-01-16 14:18:59 -05:00
David Blewett 957aef1fb1 Add Kafka 3.6 to the integration test matrix. 2024-01-11 17:51:01 -05:00
David Blewett 85539acd38
Merge pull request #636 from getsentry/base-consumer 
Pass BaseConsumer to ConsumerContext::rebalance
 2024-01-11 17:36:05 -05:00
David Blewett 6154c84a7c Release v0.36.1. 2024-01-11 17:35:00 -05:00
David Blewett 9b44abacae
Merge pull request #644 from fede1024/scanterog/return-none-on-rebalance 
Return back to the caller on rebalance events
 2024-01-11 17:24:14 -05:00
Samuel Cantero d6ec6e1596 Return back to the caller on rebalance events 
This does not affect the StreamConsumer or any other wrapper consumer.
It will only incur on an extra Poll call when there's a rebalance event.

When using bindings built upon the rust-rdkafka ffi, the caller is
responsible for initiating the rebalance calls (*assign).
If a high timeout is specified, the rebalance handler will only be
triggered once the timeout period has elapsed.

This fixes it by always returning on rebalance events except when
Timeout::Never. Poll calls with timeout::Never are expected to return
a message.
 2024-01-11 16:35:43 -05:00
David Blewett 738590b9f0
Merge pull request #645 from fede1024/davidblewett/tune-docker-compose 
Disable valgrind in CI temporarily
 2024-01-11 16:34:48 -05:00
David Blewett 321c04078a Avoid topic pollution by prefixing with test name. 2024-01-11 15:30:10 -05:00
David Blewett e347f9aece Disable valgrind for now, and start up kafka/zk before starting building 2024-01-11 14:58:07 -05:00
David Blewett 8f37c49553 Use a short timeout instead of now_or_never. 2024-01-11 14:57:47 -05:00
Sebastian Zivota 95f6cd8431 Fix simple_consumer example 2024-01-08 11:46:57 +01:00
Arpad Borsos 15044da8d9
Use `CStr::to_string_lossy` in Base{Consumer,Producer} 
In some error cases, the `Base{Consumer,Producer}` were eagerly copying strings,
and `unwrap`ing utf8 validation, just to print an error message.

This will avoid the allocation in the common case, and be panic-safe in the presumably unreachable case of invalid utf-8.
 2023-12-19 12:30:11 +01:00
Sebastian Zivota c811175785 Pass BaseConsumer to ConsumerContext::rebalance 2023-11-28 16:43:56 +01:00
David Blewett 0c5c131624 Release v0.36.0. 2023-11-09 17:25:01 -05:00
David Blewett c719e55e02
Merge pull request #617 from fede1024/scanterog/event-based-client 
Use rdkafka event API instead of the callback API
 2023-11-08 09:22:14 -05:00
Samuel Cantero c87c1e7eb8
Merge pull request #624 from fede1024/scanterog/instructions 
Update release notes
 2023-11-07 17:36:22 -03:00
Samuel Cantero 978c9649be Fix op timeout computation logic on poll_queue 2023-11-07 15:49:49 -03:00
Samuel Cantero 3b13940450 Propagate fatal errors 
With the Event API we propagate generic client instance-level errors,
such as broker connection failures, authentication issues, etc.

However, fatal errors are also propagated via the Event API. These
indicates that the particular instance of the client (producer/consumer)
becomes non-functional.
 2023-11-07 15:49:49 -03:00
Samuel Cantero 7202e7bcc1 Use closed and close_queue methods on drop 2023-11-07 15:49:49 -03:00
Samuel Cantero 34fc3356a9 Expose a close_queue and closed methods 
If you have a consumer wrapping this one (FFI cases), the outer consumer must close
the queue and serve the events via Poll. Otherwise it will hang forever
as prior to calling close there's a rebalance & rdkafka awaits a
response before continuing.
 2023-11-07 15:49:49 -03:00
Samuel Cantero 4fb2266bb9 Do not panic on transient errors on test_consume_partition_order 2023-11-07 15:49:49 -03:00
Samuel Cantero 2af36710b9 Allow creating a consumer without group.id 
Currently if a group.id is not specified we allow the use of the consumer
for fetching metadata and watermarks. Keeping this behaviour.
 2023-11-07 15:49:49 -03:00
Samuel Cantero 3b98f95686 Adapt consumer close to the event api 2023-11-07 15:49:48 -03:00
Samuel Cantero bb2aee0d45 Adapt commit_transaction to the event api 2023-11-07 15:49:48 -03:00
Samuel Cantero 74ff52a176 Propagate errors for the consumer 
rdkafka reports consumer errors via RD_KAFKA_EVENT_ERROR but producer errors gets
embedded on the ack returned via RD_KAFKA_EVENT_DR. Hence we need to return this event
for the consumer case in order to return the error to the user.
 2023-11-07 15:49:48 -03:00
Samuel Cantero c7f83a8d0d Cargo clippy 2023-11-07 15:49:48 -03:00
Samuel Cantero 54893abffc Poll should not return None if timeout has not been reached 
If timeout::Never is used, poll should eventually return a Message or
Error rather than None when handling other events like stats, rebalances,
etc.
 2023-11-07 15:49:48 -03:00
Samuel Cantero 6c8c5f0dc5 Consumer needs to read from earliest otherwise consumer will never read anything 2023-11-07 15:49:48 -03:00
Samuel Cantero f3173d57fa StreamConsumer Stream impl fixes for the event API 2023-11-07 15:49:48 -03:00
Samuel Cantero 64d2e3200f Replace poll with flush on baseProducer drop 
One poll call might not be enough to serve the delivery report
callbacks of the purged messages. The current flush impl will call
poll multiple times until the queue is empty or timeout.
 2023-11-07 15:49:48 -03:00
Samuel Cantero 32b0d249a5 Update doc comment for BorrowedMessage::from_dr_event 2023-11-07 15:49:48 -03:00
Samuel Cantero 0b885a50a0 Add comment for no-op method used on RDKafkaMessage impl of the KafkaDrop trait 2023-11-07 15:49:48 -03:00
Samuel Cantero 0a36b3d57c Explain why the TPL need to be manuallyDrop on the consumer events handler 2023-11-07 15:49:48 -03:00
Samuel Cantero 438af770ea Adapt producer Flush to the Event API semantics 2023-11-07 15:49:48 -03:00
Samuel Cantero 941cd32980 Use Arc for events in BorrowMessage 
This is required as multiple write acks are tied to a single event.
 2023-11-07 15:49:48 -03:00
David Blewett 39dec2837e Work on supporting StreamConsumer via lifetimes instead of Arc. 
The `Arc<BaseConsumer<C>>` to `MessageStream` isn't necessary anymore,
and the changes to `split_partition_queue` can be reverted as well I
think.
 2023-11-07 15:49:48 -03:00
David Blewett ebe7e9cd3b Refactor to use references and lifetimes rather than Arc. 2023-11-07 15:49:48 -03:00
David Blewett b897ec949c Pass arc by value rather than reference and fix generic type. 2023-11-07 15:49:48 -03:00
Samuel Cantero b527a3ee64 Adapt the StreamConsumer to poll the underlying BaseConsumer 2023-11-07 15:49:48 -03:00
Samuel Cantero 19f32bf824 Move to Event-based API 2023-11-07 15:49:48 -03:00
Samuel Cantero 3e9ebef32e Update release notes 2023-11-07 15:47:51 -03:00
Samuel Cantero d425a60f9c Release v0.35.0 2023-11-07 15:38:49 -03:00
Samuel Cantero 573a02a217 Release rdkafka-sys v4.7.0+2.3.0 2023-11-07 15:33:02 -03:00
Devin c4c1d8c7b0
Derives `serde::Serialize` on `Statistics` (#616) 
I realize that there is a `stats_raw` trait method, but it would be better to not have to re-type out the statistics fields myself. This came about because I would like to act on many of the individual fields locally via some aggregations, but then output them back out (along with other fields in a larger status struct) as a JSON-encoded string, e.g.:

```rs
#[derive(Serialize)]
pub struct Status {
  pub sample_field: u64,
  pub sample_field_2: boolean,
  pub statistics: Statistics,
}
```
 2023-11-07 15:25:19 -03:00
David Blewett b3f3201f29
Merge pull request #623 from fede1024/davidblewett/add-msrv-rust-version 
Add rust-version, so that our MSRV is enforced by cargo:
 2023-11-06 10:09:55 -05:00
David Blewett d12dd2d12e Add rust-version, so that our MSRV is enforced by cargo: 
https://doc.rust-lang.org/cargo/reference/manifest.html#the-rust-version-field
 2023-11-06 10:09:24 -05:00
David Blewett b125267e70
Merge pull request #622 from fede1024/davidblewett/librdkafka-2.3.0 
Bump librdkafka to 2.3.0.
 2023-11-04 08:11:17 -04:00