958 B
Events grouping
This lambda function exists as an optional add-on for Oxbow when there are
large number of events being generated by S3 Event Notifications which share a
common prefix. For example, if an external system produces a substantial number
of PutObject calls for .parquet files with the prefix
databases/external/dw_sync/ this can result in the oxbow-lambda being
inveoked with high amounts of concurrency for the same Delta table, leading to
lock contention in AWS S3.
THe events grouping function aims to address this issue by acting as an
intermediary on the events pipeline and translating unordered SQS messages from
its trigger, into FIFO messages with a specific group identifier for an SQS
FIFO destination. By grouping on the prefix (e.g.
databases/external/dw_sync/) the oxbow-lambda can then be triggered from
the SQS FIFO queue where concurrent function invocations will have a far lower likelihood for lock contention.