R Tyler Croy
d26d27950c
Playing around with reading the file from S3
...
This structure I don't like, I need to think a bit more about how to structure
the processor code
2021-06-13 09:43:26 -07:00
R Tyler Croy
2667088175
Properly build with the deltalake dependency and my features branch
2021-06-12 15:21:09 -07:00
R Tyler Croy
559c977798
Tinkering around but consuming messages from SQS in standalone mode
...
Needed to do some manual testing first to ensure that this would properly
consume from SQS and handle the events properly
2021-06-12 12:22:24 -07:00
R Tyler Croy
5f10b2d86d
Disable the deltalake dependency and pin rusoto to rustls
...
Native TLS always becomes a huge pain in the ass when building lambda functions.
Rustls is more reliable of a build, but it seems that I need to thread that
`rustls` feature through the deltalake crate
2021-06-12 10:24:14 -07:00
R Tyler Croy
2f5f68b35c
Load environment variables optionally from .env
...
This is most helpful for local testing 😸
2021-06-12 09:48:34 -07:00
R Tyler Croy
fe76b31955
Properly compile the lambda feature in
...
This commit also removes some unnecessary code which I decided was no longer of
use since S3 Event Notifications can have filters described on them upstream,
reducing the work that is necessary by Delta S3 Loader
2021-06-11 15:42:47 -07:00
R Tyler Croy
c20997b197
Sketching out the design and implementation of Delta S3 Loader
...
There is still more to think about but this is a reasonable checkpoint to commit
and push for others to look at.
2021-06-11 11:50:22 -07:00
R Tyler Croy
88264d83c9
Update the README to make it clear that this isn't going to be lambda only
2021-06-10 10:59:15 -07:00
R Tyler Croy
fb0582e4ca
Make lambda an optional build target
...
This will help for use-cases where running AWS Lambda is cost-prohibitive
because the volume of inbound events is so high
2021-05-21 19:40:59 -07:00
R Tyler Croy
3b598da87f
Ensure that the MatchedFile information gets carried out of match_record
...
Since the function is already doing some parsing, might as well carry out the
important data.
2021-04-21 14:06:24 -07:00
R Tyler Croy
4d1951c716
Implement the core of the matching code to ensure that the desired record matches
...
This will ensure that the record matches the right Source configuration entry
such that we can proceed with reading/writing of the actual data
2021-04-21 13:49:40 -07:00
R Tyler Croy
3386acc3b6
Tinkering with wiring through the events into a processing/matching loop
...
This will help filter out events which are not important
2021-04-21 11:05:19 -07:00
R Tyler Croy
95109ae12d
Add some properly structured test data and a destination audit_log delta table
2021-04-21 10:30:35 -07:00
R Tyler Croy
4fe1319e27
Working on the means of actually configuring the desired behavior
2021-04-17 15:37:47 -07:00
R Tyler Croy
922a08efdc
Returning to schema inference work
2021-04-17 14:41:18 -07:00
R Tyler Croy
a795083129
Checkpointing some work on the lambda side of things, processing inbound events
...
At this point I'm still spending a lot of time reading through some
kafka-delta-ingest and arrow code to figure out exactly how I want to do schema
inference and writes.
2021-04-14 22:21:09 -07:00
R Tyler Croy
bf1fb592e3
Set up the right dependencies and bring the bits of a lambda_runtime example in
2021-04-14 20:11:07 -07:00
R Tyler Croy
316c79fe1c
Initial commit
2021-04-14 20:00:27 -07:00