rtyler/otto
rtyler
/
otto
1
1
Fork 0
Code Issues Pull Requests Releases Activity
358 Commits 2 Branches 0 Tags 1.5 MiB
Commit Graph

358 Commits

Author SHA1 Message Date
R Tyler Croy 394a9c9ddd Implement a simplistic object store for putting artifacts 2020-11-01 14:37:13 -08:00
R Tyler Croy 1baebf061c Move the services into their own own directory for a cleaner root tree 
There's going to be a number of services..
 2020-10-30 20:05:44 -07:00
R Tyler Croy 2f8740665a I'm too tired to makefile 2020-10-28 21:03:05 -07:00
R Tyler Croy d6f99df0eb Make this makefile check for the actual file we need 2020-10-28 20:56:41 -07:00
R Tyler Croy 2b6f978313 Ensure that shunit2 exists before trying to run the acceptance tests 2020-10-28 19:38:47 -07:00
R Tyler Croy d3a618e071 Drop unnecessary checks from the GitHub Actions and try to invoke integration tests 
Really looking forward to Otto being self-hosted, but for now this will have to
do!
 2020-10-28 19:31:56 -07:00
R Tyler Croy 19dc4d3de2 Update the step library RFC to include some recent developments 2020-10-28 19:31:05 -07:00
R Tyler Croy 868f4b248e cargo fmt 2020-10-28 19:24:52 -07:00
R Tyler Croy 7073d18731 Wire through endpoints into the invocation file format 
Right now there's nothing really being put into this. I need to consider some
more details about how the steps should interact with these endpoints to fully
flesh out the Endpoint struct
 2020-10-28 16:00:14 -07:00
R Tyler Croy c9596776d2 cargo fmt 2020-10-28 15:37:18 -07:00
R Tyler Croy 99b32b86bd Update the archive step to actually generate tarballs when necessary 2020-10-28 15:35:05 -07:00
R Tyler Croy c7d290d057 Disallow archiving of files outside of the current tree 2020-10-28 14:55:45 -07:00
R Tyler Croy 9b14541909 Add some integration tests for the archive-step to cover some negative failure cases 
This still would allow archiving ../../../etc/passwd, but I'll fix that shortly
😸
 2020-10-28 13:31:52 -07:00
R Tyler Croy 65733d5437 Add shunit2 for step acceptance tests 2020-10-28 12:55:37 -07:00
R Tyler Croy 87277d8ee0 Scaffolding for the archive step. 
This includes the method stubs for the archive step. Before I continue I need to
do some thinking about how naming of archives might work, or where they can be
put for now.
 2020-10-28 11:36:53 -07:00
R Tyler Croy bc1968ffc8 Ensure that block scoped steps are propagating failures/successes upwards 
Propagating is a word I never spell correctly on the first try
 2020-10-28 10:59:14 -07:00
R Tyler Croy d821bfdc60 cargo fmt 2020-10-28 10:46:04 -07:00
R Tyler Croy 5cae7d74db Clean up some left over build warnings to shorten the compile logs 2020-10-28 10:45:41 -07:00
R Tyler Croy 1001ee6be8 Tidy up the invocation file format to actually follow the spec that I wrote 🤦 
This consolidates the run loop to use the same basic structure that the steps
will use to deserialize, which is valuable
 2020-10-28 10:44:46 -07:00
R Tyler Croy 842248b29e Convert the error-step to a simple simple simple rust program 2020-10-28 10:28:25 -07:00
R Tyler Croy 05139a9fbe Properly exit the pipeline on the agent when a step fails (non-zero return status) 2020-10-28 10:20:42 -07:00
R Tyler Croy 90e98d5522 Refactor some of the common step code into the ottoagent::step module 
This should make it easier to spin up new steps and share common behaviors
 2020-10-28 10:08:11 -07:00
R Tyler Croy 1bf172490a Implement the error step which will terminate the pipeline 
This isn't yet changing the status of the pipeline however
 2020-10-25 15:22:38 -07:00
R Tyler Croy a37d459e22 Wire the control messages into the runloop handler in the agent 
I'm *very* not pleased with how this code fits together, but it's a start and
it's _working_ (see below).

Now to wire this into a step!

    ❯ RUST_LOG=debug STEPS_DIR=$PWD/tmp ./target/debug/otto-agent ./test-pipeline.yml
     INFO  ottoagent::control > Starting the agent control server
     INFO  tide::listener::unix_listener > Server listening on http+unix:///home/tyler/source/git/otto/agent.sock
     WARN  ottoagent                     > unknown/manifest.yml does not exist, step cannot execute
    StepStart { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b }
     INFO  tide::log::middleware         > <-- Request received
     DEBUG ottoagent::control            > Receiving control request on agent sock: Terminate
     INFO  tide::log::middleware         > --> Response sent
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "-rw-r--r--  1 tyler users 1.1K Feb 20  2020 system.dot", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "-rw-r--r--  1 tyler users  43K Feb 20  2020 system.png", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "drwxr-xr-x  7 tyler users 4.0K Oct 17 15:25 target", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "-rw-r--r--  1 tyler users 1.8K Oct 25 14:46 test-pipeline.yml", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "drwxr-xr-x  5 tyler users 4.0K Oct 22 20:35 tmp", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "+ ls -lah", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "+ tail -n 5", stream: Stdout }
    StepOutput { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b, buffer: "+ sleep 15", stream: Stdout }
    StepEnd { symbol: "sh", uuid: 5599cffb-f23a-4e0f-a0b9-f74654641b2b }
     DEBUG ottoagent                     > Processing control message in runloop: Terminate
     INFO  ottoagent                     > Runloop has been asked to terminate, exiting
 2020-10-25 14:52:43 -07:00
R Tyler Croy f7436c111e Receive control requests, not currently doing anything with them however. 
Tested with:
    ❯ curl --unix-socket $PWD/agent.sock -X POST -d '{"type":"Terminate"}' http://agent/control
    {}
 2020-10-25 14:28:42 -07:00
R Tyler Croy 3498697411 Incorporate tide into the agent for a primitive HTTP over unix socket control mechanism 
HTTP is heavyweight but it's also Sofa King easy.
 2020-10-25 14:16:29 -07:00
R Tyler Croy 60f2a7745f Refactor some of the agent manifest loading to be ... sorta testable 2020-10-24 16:10:36 -07:00
R Tyler Croy 47676dab4e cargo fmt 2020-10-24 15:39:29 -07:00
R Tyler Croy 67aa57dcc9 Emit structured logs from the agent to help isolate which steps do what 
This is stupidly dumping to stdout at the moment, but the raw structure is
there.
 2020-10-24 15:35:57 -07:00
R Tyler Croy 8aa82f58d4 Add a number of uuids to the pipeline data structure for tracking different actions 
These are going to be known in the future in order to appropriately associate
logs, etc with different parts of the pipeline
 2020-10-23 20:30:12 -07:00
R Tyler Croy c11c05aac0 Relocate the primitive agent into the agent directory and wire in block scoped steps for the dir step 
This was WAY EASIER than I thought it would be, which has me extra pumped about
this design decision.

The pipeline.yml sent to the agent:

  ---
  steps:
    - symbol: sh
      parameters:
        script: 'ls -lah | tail -n 5'

    - symbol: sh
      parameters:
        script: 'echo "Hello world from a script"'

    - symbol: dir
      parameters:
        directory: 'stdlib'
        block:
          - symbol: sh
            parameters:
              script: 'pwd && ls -lah'
          - symbol: echo
            parameters:
              message: 'Hello from a block-scoped step!'

    - symbol: unknown
      parameters:
        message: 'this should fail'

Then executing it in the tree:

  ❯ STEPS_DIR=$PWD/tmp ./target/debug/otto-agent ./test-pipeline.yml
  sh exists
  sh exists
  dir exists
  unknown/manifest.yml does not exist, step cannot execute
  NORMALLY THIS WOULD ERROR BEFORE ANYTHING EXECUTES
  ---
  -rw-r--r--  1 tyler users 1.1K Feb 20  2020 system.dot
  -rw-r--r--  1 tyler users  43K Feb 20  2020 system.png
  drwxr-xr-x  7 tyler users 4.0K Oct 17 15:25 target
  -rw-r--r--  1 tyler users 1.5K Oct 22 20:38 test-pipeline.yml
  drwxr-xr-x  5 tyler users 4.0K Oct 22 20:35 tmp
  Hello world from a script
  sh exists
  echo exists
  ---
  /home/tyler/source/git/otto/stdlib
  total 20K
  drwxr-xr-x  5 tyler users 4.0K Oct 22 20:02 .
  drwxr-xr-x 22 tyler users 4.0K Oct 22 20:38 ..
  drwxr-xr-x  4 tyler users 4.0K Oct 22 20:19 dir
  drwxr-xr-x  2 tyler users 4.0K Oct 22 20:28 echo
  drwxr-xr-x  4 tyler users 4.0K Oct 21 19:40 sh
  Hello from a block-scoped step!

Voila!
 2020-10-22 20:35:49 -07:00
R Tyler Croy 43116fdd5a Scaffolding for the dir step 
I need to refactor the agent code into a crate before continuing here
 2020-10-22 20:11:16 -07:00
R Tyler Croy dabf335641 Add nanomsg which will be used for agent<->step IPC 2020-10-22 20:01:55 -07:00
R Tyler Croy d9194eaae3 Fix heading level 2020-10-21 20:57:42 -07:00
R Tyler Croy afdbae72ad Add sketch of what the agent control port and some messages might look like 
This is still in the research/prototyping phase obviously I'm currently thinking
of nanomsg for the IPC mechanism since that may offer the best cross-platform
support for sending JSON back and forth between processes in a lightweight
manner
 2020-10-21 20:54:48 -07:00
R Tyler Croy c5fc6401a9
Merge pull request #29 from rtyler/step-libraries 
Mainlining the work in progress on step libraries
 2020-10-21 19:39:35 -07:00
R Tyler Croy a5de9294aa Implement the simplest of runners using the step libraries 
Using the following test pipeline file:

  # This file is just a pretend primitive test pipeline with a bunch of steps
  ---
  steps:
    - symbol: sh
      parameters:
        script: 'ls -lah | tail -n 5'
    - symbol: sh
      parameters:
        script: 'echo "Hello world from a script"'
    - symbol: unknown
      parameters:
        message: 'this should fail'

And then invoking as such:

  ❯ STEPS_DIR=tmp ./target/debug/primitive-agent test-pipeline.yml
  sh exists
  sh exists
  unknown/manifest.yml does not exist, step cannot execute
  NORMALLY THIS WOULD ERROR BEFORE ANYTHING EXECUTES
  ---
  entry: "tmp/sh/sh-step"
  -rw-r--r--  1 tyler users 1.1K Feb 20  2020 system.dot
  -rw-r--r--  1 tyler users  43K Feb 20  2020 system.png
  drwxr-xr-x  7 tyler users 4.0K Oct 17 15:25 target
  -rw-r--r--  1 tyler users  304 Oct 18 15:04 test-pipeline.yml
  drwxr-xr-x  4 tyler users 4.0K Oct 17 16:07 tmp
  entry: "tmp/sh/sh-step"
  Hello world from a script

So at a _very_ _very_ primitive level the concept is working 👏
 2020-10-18 15:05:05 -07:00
R Tyler Croy 3760508cb8 Merge remote-tracking branch 'origin/main' into step-libraries 2020-10-18 13:51:57 -07:00
R Tyler Croy 993a6515d2 Missed these in the commit, oops 2020-10-18 13:49:39 -07:00
R Tyler Croy e79a6505c9 Add a simple github action for rust 
Also ran cargo fmt to avoid issues
 2020-10-18 13:48:55 -07:00
R Tyler Croy 8ff38597e3 Disable the eventbus-inmemory build for now, the front fell off 2020-10-18 13:41:43 -07:00
R Tyler Croy 8ea6eba6cc Starting to implement a very very primitive agent 
This is not something that will ever be used, but I need a test scaffold for
step libraries
 2020-10-18 13:38:24 -07:00
R Tyler Croy f53dbb223e Fix makefile typo 2020-10-18 13:37:44 -07:00
R Tyler Croy 9cc30b3eef Add some Makefile silliness to build all the stdlib step libraries 
These are currently just being dumped into the PWD since osp doesn't yet
know anything about command line arguments
 2020-10-18 13:37:44 -07:00
R Tyler Croy c1d9ce6010 Add the echo step, which is a primitive example of a multiarch step library 2020-10-18 13:37:44 -07:00
R Tyler Croy d820a75ed5 Include the updated manifest spec in the rfc 2020-10-18 13:37:44 -07:00
R Tyler Croy d7c8a1a6f5 Add some release binary size optimizations to the root Cargo.toml 
Every step library should be stripped and shrinkeydinked as much as possible
 2020-10-18 13:37:44 -07:00
R Tyler Croy 132b340419 Make sure osp can easily include files from the step library directory and outside 2020-10-18 13:37:44 -07:00
R Tyler Croy 5048c04bf9 Implement the osp tool, something simple to package a step library into a tarball 2020-10-18 13:37:43 -07:00
R Tyler Croy ba6995a805 Minor rust cleanup for the sh step 2020-10-18 13:34:14 -07:00