R Tyler Croy
d26d27950c
Playing around with reading the file from S3
...
This structure I don't like, I need to think a bit more about how to structure
the processor code
2021-06-13 09:43:26 -07:00
R Tyler Croy
2667088175
Properly build with the deltalake dependency and my features branch
2021-06-12 15:21:09 -07:00
R Tyler Croy
559c977798
Tinkering around but consuming messages from SQS in standalone mode
...
Needed to do some manual testing first to ensure that this would properly
consume from SQS and handle the events properly
2021-06-12 12:22:24 -07:00
R Tyler Croy
5f10b2d86d
Disable the deltalake dependency and pin rusoto to rustls
...
Native TLS always becomes a huge pain in the ass when building lambda functions.
Rustls is more reliable of a build, but it seems that I need to thread that
`rustls` feature through the deltalake crate
2021-06-12 10:24:14 -07:00
R Tyler Croy
2f5f68b35c
Load environment variables optionally from .env
...
This is most helpful for local testing 😸
2021-06-12 09:48:34 -07:00
R Tyler Croy
c20997b197
Sketching out the design and implementation of Delta S3 Loader
...
There is still more to think about but this is a reasonable checkpoint to commit
and push for others to look at.
2021-06-11 11:50:22 -07:00
R Tyler Croy
fb0582e4ca
Make lambda an optional build target
...
This will help for use-cases where running AWS Lambda is cost-prohibitive
because the volume of inbound events is so high
2021-05-21 19:40:59 -07:00
R Tyler Croy
4fe1319e27
Working on the means of actually configuring the desired behavior
2021-04-17 15:37:47 -07:00
R Tyler Croy
a795083129
Checkpointing some work on the lambda side of things, processing inbound events
...
At this point I'm still spending a lot of time reading through some
kafka-delta-ingest and arrow code to figure out exactly how I want to do schema
inference and writes.
2021-04-14 22:21:09 -07:00
R Tyler Croy
bf1fb592e3
Set up the right dependencies and bring the bits of a lambda_runtime example in
2021-04-14 20:11:07 -07:00
R Tyler Croy
316c79fe1c
Initial commit
2021-04-14 20:00:27 -07:00
R Tyler Croy