scribd
/
objinsync
mirror of https://github.com/scribd/objinsync
1
0
Fork 0
Code Issues Releases Wiki Activity
47 Commits 8 Branches 16 Tags 193 KiB
Go 97.7%
Makefile 1.2%
Dockerfile 1.1%
Go to file
Oleh Motrunych 6da3fab377
Pe 4649 snyk issues fix (#30) 
* upgraded packages - working on snyk detected issues

* updated version

* added missing packages

* updated Docker file
 		2023-10-19 10:59:44 -07:00
.github/workflows Pe 4649 snyk issues fix (#30) 2023-10-19 10:59:44 -07:00
pkg/sync Purge uidCache after rm file (#14) 2021-04-22 16:29:52 -07:00
.dockerignore setup none-root user in final image stage 2020-08-27 11:48:04 -07:00
.gitignore add release step to github action 2020-03-02 16:16:42 -08:00
.goreleaser.yml release arm binaries 2021-06-07 20:28:26 -07:00
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md 2020-03-09 11:04:07 -07:00
Dockerfile Pe 4649 snyk issues fix (#30) 2023-10-19 10:59:44 -07:00
LICENSE Add the MIT license 2020-03-02 05:42:48 -08:00
Makefile add watch make target 2020-04-03 14:39:24 -07:00
README.md Fixing small typo in README.md (#17) 2021-10-09 17:09:07 -07:00
go.mod Pe 4649 snyk issues fix (#30) 2023-10-19 10:59:44 -07:00
go.sum Pe 4649 snyk issues fix (#30) 2023-10-19 10:59:44 -07:00
main.go Add interval flag for configuring the pull interval (#11) 2021-04-15 23:29:55 -07:00

README.md

ObjInSync

CI/CD

Daemon to continuously and incrementally synchronize a directory from remote object store to a local directory.

Usage

objinsync pull --exclude '**/__pycache__/**' s3://bucket/keyprefix ./localdir

When running in daemon mode (without --once flag), a health check endpoint is served at :8087/health and a prometheus metrics endpoint is served at :8087/metrics. You can use --status-addr to override the binding address.

Objinsync also comes with builtin Sentry integration. To enable it, set the SENTRY_DSN environment variable.

You can also run objinsync in pull once mode, which behaves just like aws s3 sync:

objinsync pull --once s3://bucket/keyprefix ./localdir

To use with Minio instead of S3, you can set --s3-endpoint and --disable-ssl flags for pull command as you see fit.

The -i or --interval flags allows to configure the pull time interval, which is 5 seconds by default:

objinsync pull --interval 20s s3://bucket/keyprefix ./localdir

Enable debug logs by setting the DEBUG environment variable DEBUG=1 objinsync pull ...

Installation

Simply download the prebuilt single binary from release page or use go get command:

go get github.com/scribd/objinsync

Pre-built docker images are available at https://github.com/orgs/scribd/packages/container/package/objinsync.

Development

Run tests

make test

Run from source

AWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags

To cut a release, push tag to remote in the format of vx.x.x.