R Tyler Croy
e3f6e7787a
|
||
|---|---|---|
| common | ||
| config/bundled | ||
| contrib | ||
| develop | ||
| migrations | ||
| tests | ||
| web | ||
| workers/kube-provisioner | ||
| .gitignore | ||
| Cargo.toml | ||
| LICENSE | ||
| Makefile | ||
| README.adoc | ||
| base.mk | ||
README.adoc
placementd
The placementd service acts as a intermediary between the Neutron compute
plane and third party orchestration tools like
Apache Airflow.
Documentation
Development
The full development stack for placementd requires Kubernetes, which for
local development purposes can be accomplished with
kind. Launching the cluster should be done
with: kind cluster create --config contrib/kind.yml
Once kind is up and running, there are a number of make targets defined in
the Makefile to help with development, simply running make will list the
built-in help for these targets. Generally speaking make migrations develop
will ensure that the development database (PostgreSQL running in kind) is
configured and deployed.
For simply running tests, cargo test can run the unit tests, while make
check can run all unit and integration tests.
Environment Variables
| Name | Default | Description |
|---|---|---|
|
|
IP address and port to bind the HTTP/API server to |
|
Full database connection string (e.g. |
|
|
Full URL for loading configuration, must be a URL scheme supported by object_store |
|
|
An optional SNS topic to be notified of for configuration changes |
|
|
URL to the Kubernetes API that the placementd |
|
|
Logging level to set, e.g. |
Design
Database task system
Task States
-
planned: default task state, all new submissions are in this state -
provisioning: task has been submitted to a provisioner (e.g. Kubernetes) -
running: task has started running in the provisioner, in the Kubernetes example this means that the Spark cluster is online and running the job -
completed: the Spark job has completed but its resources/reporting has not yet finished. -
finalized: all resources associated with the task have been deprovisioned. -
When a runsubmit API call is is received a new task is created in the database: state = PLANNED
-
SubmitWorker does a SELECT * FROM tasks WHERE state = PLANNED FOR UPDATE LIMIT 1
-
submits the Deployment to EKS
-
busy-waits for the Deployment to be "Ready"
-
submits the job via REST to the manager node
-
updates the state to RUNNING
-
StatusWorker does a SELECT * FROM tasks WHERE state = RUNNING FOR UPDATE LIMIT 1
-
StatusWorker interrogates the REST API on the manager, for its status:
-
if it’s still running: then exit
-
if it’s finished: set state to COMPLETED
-
CleanupWorker does a SELECT * FROM tasks WHERE state = COMPLETED FOR UPDATE LIMIT 1
-
Deletes resources in EKS
-
Once resources are complete, set state to FINALIZED