Commit Graph

18 Commits

Author SHA1 Message Date
R Tyler Croy d26d27950c Playing around with reading the file from S3
This structure I don't like, I need to think a bit more about how to structure
the processor code
2021-06-13 09:43:26 -07:00
R Tyler Croy 2667088175 Properly build with the deltalake dependency and my features branch 2021-06-12 15:21:09 -07:00
R Tyler Croy 559c977798 Tinkering around but consuming messages from SQS in standalone mode
Needed to do some manual testing first to ensure that this would properly
consume from SQS and handle the events properly
2021-06-12 12:22:24 -07:00
R Tyler Croy 5f10b2d86d Disable the deltalake dependency and pin rusoto to rustls
Native TLS always becomes a huge pain in the ass when building lambda functions.
Rustls is more reliable of a build, but it seems that I need to thread that
`rustls` feature through the deltalake crate
2021-06-12 10:24:14 -07:00
R Tyler Croy 2f5f68b35c Load environment variables optionally from .env
This is most helpful for local testing 😸
2021-06-12 09:48:34 -07:00
R Tyler Croy fe76b31955 Properly compile the lambda feature in
This commit also removes some unnecessary code which I decided was no longer of
use since S3 Event Notifications can have filters described on them upstream,
reducing the work that is necessary by Delta S3 Loader
2021-06-11 15:42:47 -07:00
R Tyler Croy c20997b197 Sketching out the design and implementation of Delta S3 Loader
There is still more to think about but this is a reasonable checkpoint to commit
and push for others to look at.
2021-06-11 11:50:22 -07:00
R Tyler Croy 88264d83c9 Update the README to make it clear that this isn't going to be lambda only 2021-06-10 10:59:15 -07:00
R Tyler Croy fb0582e4ca Make lambda an optional build target
This will help for use-cases where running AWS Lambda is cost-prohibitive
because the volume of inbound events is so high
2021-05-21 19:40:59 -07:00
R Tyler Croy 3b598da87f Ensure that the MatchedFile information gets carried out of match_record
Since the function is already doing some parsing, might as well carry out the
important data.
2021-04-21 14:06:24 -07:00
R Tyler Croy 4d1951c716 Implement the core of the matching code to ensure that the desired record matches
This will ensure that the record matches the right Source configuration entry
such that we can proceed with reading/writing of the actual data
2021-04-21 13:49:40 -07:00
R Tyler Croy 3386acc3b6 Tinkering with wiring through the events into a processing/matching loop
This will help filter out events which are not important
2021-04-21 11:05:19 -07:00
R Tyler Croy 95109ae12d Add some properly structured test data and a destination audit_log delta table 2021-04-21 10:30:35 -07:00
R Tyler Croy 4fe1319e27 Working on the means of actually configuring the desired behavior 2021-04-17 15:37:47 -07:00
R Tyler Croy 922a08efdc Returning to schema inference work 2021-04-17 14:41:18 -07:00
R Tyler Croy a795083129 Checkpointing some work on the lambda side of things, processing inbound events
At this point I'm still spending a lot of time reading through some
kafka-delta-ingest and arrow code to figure out exactly how I want to do schema
inference and writes.
2021-04-14 22:21:09 -07:00
R Tyler Croy bf1fb592e3 Set up the right dependencies and bring the bits of a lambda_runtime example in 2021-04-14 20:11:07 -07:00
R Tyler Croy 316c79fe1c Initial commit 2021-04-14 20:00:27 -07:00