buoyant-data
/
oxbow
1
0
Fork 0
Code Pull Requests Releases Activity
oxbow/lambdas/webhook
History
R Tyler Croy 6d1bc34b86 Introduce the bulk of sqs-ingest with some refactorings for the webhook 
The webhook and sqs-ingest lambdas both effectively need to take strings
of data and append them to a configured Delta Lake table, so the shared
code comes "up" into the oxbow crate
 		2024-04-08 08:40:05 -07:00
..
src Introduce the bulk of sqs-ingest with some refactorings for the webhook 2024-04-08 08:40:05 -07:00
.gitignore Add a webhook lambda for appending JSONL 2024-03-12 11:10:42 -07:00
Cargo.toml Properly checkpoint writes via the webhook lambda 2024-04-01 11:12:32 -07:00
README.adoc Update the webhook documentation with important settings for use 2024-04-05 15:42:43 -07:00

README.adoc

<html lang="en"> <head> </head>

Webhook Lmabda

The webhook lambda can help receive JSONL formatted webhook payloads and conver those into append-only writes to pre-configured Delta tables. It is meant to be deployed once per Delta table, that is to say that if you need to do ingestion for multiple Delta tables the best option is to deploy multiple copies of the Webhook lambda, each with a different configuration.

Table of Contents

Environment Variables

Name

Default Value

Notes

RUST_LOG

error

Set the log level, e.g. info, warn, error. Can be scoped to specific modules, i.e. oxbow=debug

PRESHARED_WEBHOOK_SECRET

null

Required if this is not set the Lambda will not run properly!

DELTA_TABLE_URI

null

Required the S3 location of the Delta table which should be appended to.

AWS_S3_LOCKING_PROVIDER

Set to dynamodb to enable safe concurrent writes to the table

DYNAMO_LOCK_TABLE_NAME

Set to the DynamoDB table used for locking, required for safe concurrent writes.
Last updated 2024-05-23 19:12:47 UTC
</html>