rtyler/otto
rtyler
/
otto
1
1
Fork 0
Code Issues Pull Requests Releases Activity
358 Commits 2 Branches 0 Tags 1.5 MiB
Commit Graph

358 Commits

Author SHA1 Message Date
R Tyler Croy 7b2dfdb749 Run the agent inside a pipeline specific work directory 
This does mean that the `git` step has to re-clone every time and all work must
be repeated, but I think there are other ways to solve those problems than
sharing workspace directories between pipeline runs.

Validated with the following invocation file:

```json

{
    "pipeline": "2265b5d0-1f70-46de-bf50-f1050e9fac9a",
    "steps" : [
        {
            "symbol":  "sh",
            "uuid": "5599cffb-f23a-4e0f-a0b9-f74654641b2b",
            "context": "3ce1f6fb-79ca-4564-a47e-98265f53ef7f",
            "parameters" : {
                "script" : "pwd"
            }
        }
    ]
}
```

And invoked in tmp/ via:

    STEPS_DIR=$PWD ../target/debug/otto-agent ../test-steps.json

Fixes #33
 2020-11-19 20:14:09 -08:00
R Tyler Croy 1b3e67d885 Token commit 2020-11-16 14:32:44 -08:00
R Tyler Croy 4b6f99a099 Finish out the basic implementation of the git step 
Right now this won't support SSH URLs, but I'm okay with that.

Fixes #32
 2020-11-15 20:29:44 -08:00
R Tyler Croy 28513c7a0b Implement the most basic version of the git step which can just clone a URL 
This doesn't yet support the `ref` parameter, and is frankly pretty dumb

Fixes #32
 2020-11-15 16:12:12 -08:00
R Tyler Croy 610b16d542 Restructure the shunit2 integration tests to use a test directory for each run 
This will help avoid conflicts
 2020-11-15 16:11:32 -08:00
R Tyler Croy 0d17491f97 Make the invalid cases for the archive step fail again for the right reasons 2020-11-15 15:10:08 -08:00
R Tyler Croy 8d5d2b7cea cargo update 2020-11-15 14:45:43 -08:00
R Tyler Croy e4e3660d3a Upgrade to the latest tide release 👏 2020-11-15 14:45:23 -08:00
R Tyler Croy bea61697fa Ensure that `make test` will return a non-zero on shunit2 test failures 
GitHub Actions were in fact catching the failure. Oops
 2020-11-15 14:40:59 -08:00
R Tyler Croy 273a06ad94 Fix failing integration tests for the archive step 
These had been failing for a few commits, oops
 2020-11-15 14:39:14 -08:00
R Tyler Croy 1b6ebdd3db Work in progress on the git step 
See #32
 2020-11-15 14:39:00 -08:00
R Tyler Croy 43540a5dd6 Switch the invocation file formats from YAML to JSON 
YAML is a bonkers format with a lot of leeway for bugs, JSON is dumb but very
predictable.

Since these are machine-read files, they can be JSON. The human-editable step
library manifests will remain YAML
 2020-11-14 20:15:16 -08:00
R Tyler Croy d92a72ec7d Restructuring the models to allow for parallel execution of contexts 
This will lay the ground work for more complex pipeline verbs like fanout and
parallel

This also changes the invocation file format for the agent to basically:

    ---
    pipeline: 'some-uuid'
    steps:
      - symbol: sh
        context: 'some-uuid'
        parameters:

This means that the agents will be a little more stupid when it comes to the
overall pipeline structure
 2020-11-14 15:43:28 -08:00
R Tyler Croy bd41fecab3 Slight cleanup/refactoring 2020-11-11 19:59:16 -08:00
R Tyler Croy 414ad11f8e Add simple single line comments 2020-11-08 15:55:08 -08:00
R Tyler Croy f496748483 Shuffle the parser to make stages truly optional, allowing inline orphan steps 
This is the manifestation of @steven-terrana's feedback. Basically the pipeline
is linear from top to bottom, and the author can put un-staged steps anywhere
along the path that they want. This should make the simplest pipelines pretty
dang simple.

These steps will have to be rendered somehow by a frontend at some point, but
that's a problem for later! 😸
 2020-11-08 15:38:01 -08:00
R Tyler Croy 5c6fc665ba Remove the stages{} block which is not terribly useful. 
This starts to make room for multiple root-level verbs in the pipeline, but also
for stages to be optional, e.g.

    pipeline {
        steps {
            sh 'make all'
            archive 'build/*.tar.gz'
        }
    }

(the above doesn't parse yet)

Based on a discussion about the merits of the stage block with @steven-terrana
 2020-11-08 15:16:00 -08:00
R Tyler Croy 603ad5eb4f Add some documentation around the parser service 2020-11-08 15:09:07 -08:00
R Tyler Croy b70cf8c17b Update the object-store to also allow use with catflap 2020-11-08 15:01:06 -08:00
R Tyler Croy 24427fea68 cargo update 2020-11-08 14:53:40 -08:00
R Tyler Croy 649d29f3d2 cargo fmt 2020-11-08 14:50:54 -08:00
R Tyler Croy d5303133e8 Add a simple /health endpoint which all Otto services should have 2020-11-08 14:50:31 -08:00
R Tyler Croy 8be8dc98d7 Start returning a real pipeline object 
I need to figure out how tos share this ParsePipelineResponse model with the
models directory which is the real source of truth here
 2020-11-08 14:37:33 -08:00
R Tyler Croy 35ec2e63bb Flesh out the apispec some more and make the basic schemathesis runs pass 
This is passing only with the local patch I made too work around this issue:
    https://github.com/schemathesis/schemathesis/issues/850
 2020-11-08 14:09:35 -08:00
R Tyler Croy 25a2bf6fb6 Exploring using openapi and schemathesis to describe the parser service API 
Running via:
    schemathesis run ./services/parser/apispec.yml --base-url=http://localhost:7672 --checks all

The parser service is also using `catflap` for development: https://github.com/passcod/catflap

See #13
 2020-11-08 09:27:19 -08:00
R Tyler Croy b89e1112e0 Purge additional outdated files 2020-11-07 19:34:22 -08:00
R Tyler Croy a598519f4d Remove out-dated grammar directory. 
The new grammar can be found in services/parser
 2020-11-07 19:32:30 -08:00
R Tyler Croy 921970c8c2 Add some meeting notes from this morning 2020-11-07 14:44:54 -08:00
R Tyler Croy 3c2daa412a Add a table of contents 2020-11-04 21:22:36 -08:00
R Tyler Croy 17db1f38a5 Link to RFCs from the readme 2020-11-04 21:19:40 -08:00
R Tyler Croy 2f3fd32047 Clean up some other RFCs 2020-11-04 21:15:02 -08:00
R Tyler Croy 5b2e8a8c43 Update RFC #11 to include some more step library details 2020-11-04 21:06:45 -08:00
R Tyler Croy f7c845e374 Upgrade to async-std 1.7.0 
WOOOOO
 2020-11-04 13:05:55 -08:00
R Tyler Croy 8404b6fca2 Move the parser into the services directory where it belongs 
This commit also cleans up a lot of build warnings that were bugging me
 2020-11-03 20:54:05 -08:00
R Tyler Croy 025dac1673 Starting to service-erize the Parser 2020-11-03 20:46:02 -08:00
R Tyler Croy 64fef7a55e Add some test cases as integration tests 2020-11-03 16:12:53 -08:00
R Tyler Croy 0d1f30665b Update the grammar to parse out stage-level properties. 
Right now I'm expecting this to work for a `name` key but there may be others
that come to mind shortly
 2020-11-03 15:42:59 -08:00
R Tyler Croy a39c6f3ffd cargo fmt 2020-11-03 15:22:47 -08:00
R Tyler Croy a29500dcc6 Adjust the otto-models crate to allow for positional and keyword argument parameters for steps 
This is required to support syntaxes like:

    sh 'make build'

or

    sh script: 'make build'

Though the latter does not currently have any support in the parser
 2020-11-03 15:14:32 -08:00
R Tyler Croy ae2ac97cbb This parser code is trash, but at least is taking the pest grammar and doing something 
I'm really really not pleased with how this code is structured, and how brittle
this is. For the first pest parser I've written, I'll cut myself a break, but
this must be mostly reworked 😦
 2020-11-02 21:36:38 -08:00
R Tyler Croy 101ef3b7cc Refactor some common code out into the "models" module for sharing 
This will be shared between the parser and the agent so that everybody has the
same views on what the pipeline data structure is
 2020-11-02 21:35:59 -08:00
R Tyler Croy 189f18e791 Expanding the grammar a bit to handle the types of tokens steps need 2020-11-01 21:16:32 -08:00
R Tyler Croy 5352cd3d11 Playing around with writing a grammar with pest 
Giving up on ANTLR4 for now, pest looks promising though!
 2020-11-01 21:04:05 -08:00
R Tyler Croy 56ffbaadb2 Implement the corresponding `unarchive` step to grab a pipeline's artifacts later on 
This can be used to grab an archived artifact on another agent, for example.

Tested with:

  ---
  uuid: '2265b5d0-1f70-46de-bf50-f1050e9fac9a'

  contexts:
    - name: 'Prep'
      uuid: '3ce1f6fb-79ca-4564-a47e-98265f53ef7f'
      environment:
        CLOUD: 'yes'

    - name: 'Build'
      uuid: '5e9d4943-33e5-4945-8a97-a11421cb4b11'

  steps:
    - symbol: sh
      uuid: '2265b5d0-1f70-46de-bf50-f1050e9fac9a'
      context: '5e9d4943-33e5-4945-8a97-a11421cb4b11'
      parameters:
        script: 'echo "hello" > hello.log'

    - symbol: archive
      context: '5e9d4943-33e5-4945-8a97-a11421cb4b11'
      uuid: 'd479e450-9ada-4127-84ca-e0576d0c0c98'
      parameters:
        artifacts: 'hello.log'

    - symbol: sh
      uuid: '5599cffb-f23a-4e0f-a0b9-f74654641b2b'
      context: '3ce1f6fb-79ca-4564-a47e-98265f53ef7f'
      parameters:
        script: 'rm -f hello.log'

    - symbol: unarchive
      uuid: '2265b5d0-1f70-46de-bf50-f1050e9fac9a'
      context: '5e9d4943-33e5-4945-8a97-a11421cb4b11'
      parameters:
        name: 'hello.log'

    - symbol: sh
      uuid: '7f7ec29b-ad62-4fba-9c5f-db7a49a2d658'
      context: '3ce1f6fb-79ca-4564-a47e-98265f53ef7f'
      parameters:
        script: 'cat hello.log'
 2020-11-01 15:52:41 -08:00
R Tyler Croy 2dc7670cbd cargo fmt 2020-11-01 15:26:09 -08:00
R Tyler Croy f72823db90 👏 Implement artifact uploading from the archive step 👏 
Running the object store with: ./scripts/shoreman

❯ tree tmp/objects
tmp/objects
└── 2265b5d0-1f70-46de-bf50-f1050e9fac9a
    └── hello.log

1 directory, 1 file

The full pipeline yml that was passed to the agent below. But in essence this
demonstrates the approach for the agent to pass endpoints to the steps for their
uses

  ---
  uuid: '2265b5d0-1f70-46de-bf50-f1050e9fac9a'

  contexts:
    - name: 'Prep'
      uuid: '3ce1f6fb-79ca-4564-a47e-98265f53ef7f'
      environment:
        CLOUD: 'yes'

    - name: 'Build'
      uuid: '5e9d4943-33e5-4945-8a97-a11421cb4b11'

  steps:
    - symbol: sh
      uuid: '5599cffb-f23a-4e0f-a0b9-f74654641b2b'
      context: '3ce1f6fb-79ca-4564-a47e-98265f53ef7f'
      parameters:
        script: 'ls -lah | tail -n 5'

  #  - symbol: error
  #    uuid: '3652dfe3-a07c-4bd7-b8e5-303100a3873c'
  #    context: '3ce1f6fb-79ca-4564-a47e-98265f53ef7f'
  #    parameters:
  #      message: 'Failing out early'

    - symbol: sh
      uuid: '2265b5d0-1f70-46de-bf50-f1050e9fac9a'
      context: '5e9d4943-33e5-4945-8a97-a11421cb4b11'
      parameters:
        script: 'echo "Hello world from a script"'

    - symbol: dir
      uuid: '022c09ef-15f2-456a-a93a-92e584c1a1f4'
      context: 'a4cdad98-caf0-4705-ae1f-7986f8a1adf3'
      parameters:
        directory: 'stdlib'
        block:
          - symbol: sh
            uuid: '81e8309e-e615-49d4-946c-22527ebce8e1'
            context: 'a4cdad98-caf0-4705-ae1f-7986f8a1adf3'
            parameters:
              script: 'pwd && ls -lah'
          - symbol: echo
            uuid: '7f7ec29b-ad62-4fba-9c5f-db7a49a2d658'
            context: 'a4cdad98-caf0-4705-ae1f-7986f8a1adf3'
            parameters:
              message: 'Hello from a block-scoped step!'

    - symbol: sh
      uuid: '2265b5d0-1f70-46de-bf50-f1050e9fac9a'
      context: '5e9d4943-33e5-4945-8a97-a11421cb4b11'
      parameters:
        script: 'echo "hello" > hello.log'

    - symbol: archive
      context: '5e9d4943-33e5-4945-8a97-a11421cb4b11'
      uuid: 'd479e450-9ada-4127-84ca-e0576d0c0c98'
      parameters:
        artifacts: 'hello.log'
 2020-11-01 15:23:56 -08:00
R Tyler Croy cda0f2ff9e Refactor ottoagent::run to take the full pipeline rather than just steps 
The Pipeline struct has some valuable information needed to construct endpoints
and prepare other data for the runtime
 2020-11-01 15:02:49 -08:00
R Tyler Croy e55d2c38aa Disable the slower release profile builds for now 2020-11-01 15:02:18 -08:00
R Tyler Croy 389da374cf Remove stale services from the Procfile 
Right now auctioneer and eventbus don't really do spit, so no need to launch
them
 2020-11-01 14:37:13 -08:00
R Tyler Croy 51631ed09c Remove the processors/ which represent a slightly different take from earlier in Otto's development 
This concept is likely going to come back in some form or another, but I don't
need to muddy the tree with it at the moment
 2020-11-01 14:37:13 -08:00