rtyler/otto
rtyler
/
otto
1
1
Fork 0
Code Issues Pull Requests Releases Activity
358 Commits 2 Branches 0 Tags 1.5 MiB
Commit Graph

358 Commits

Author SHA1 Message Date
R Tyler Croy f8dc3af95e Update the cargo versions 
Fixes #64
 2021-03-03 22:04:25 -08:00
R Tyler Croy 102ded9756 Finally fiddled around enough to make this work 2021-03-03 22:04:25 -08:00
R Tyler Croy 2ebf41bb31 This is just awful 
Hopefully finding better ways in https://github.com/launchbadge/sqlx/issues/1083
 2021-03-03 22:04:25 -08:00
R Tyler Croy 14d78e009f Check pointing some exploration with graphql for the reldata service 2021-03-03 22:04:25 -08:00
R Tyler Croy 08d584974b Upgrade to tide 0.16 across the board to make the async-graphql example code work 2021-03-03 22:04:25 -08:00
R Tyler Croy 78534ca7d1 Add some Makefile targets to make working with sqlx a little more simple 2021-03-03 22:04:25 -08:00
R Tyler Croy 8a1a47669c Scaffolding up reldata service's data migrations 
Right now I'm just working within sqlite, but the migrations will need to be
mirrored for PostgreSQL in the future in `migrations/postgres`

Running this entails using the sqlx-cli (`cargo install sqlx-cli`)

    % sqlx migrate --source migrations/sqlite run
 2021-03-03 22:04:25 -08:00
R Tyler Croy f017ea4660 Pull in the async-graphql dependency for reldata 
I'm hoping that I can map objects through with graphql relatively easily so I
don't have to go full REST API for this service

See #63
 2021-03-03 22:04:25 -08:00
R Tyler Croy 4e9c851dd6 Refactor some dependencies to allow jumping to async-std 1.9.0 
There are some good stabilizations and other work in this release that I want.

Also, my Cargo.tomls have been too strict, need to start relaxing that
 2021-03-03 22:04:25 -08:00
R Tyler Croy 60e0b82247 Add a stub relational data service while I explore GraphQL 
See #63
 2021-03-03 22:04:25 -08:00
R Tyler Croy a9adeb2d8a Map the project and system configuration defined in the demo directory 2021-03-03 22:04:24 -08:00
R Tyler Croy 0412bf4226 Sketching out some system and service specific configuration 
The `demo/config` directory is eventually what I would expect to live in
`/etc/otto` on Otto hosts.

See #39 and #52
 2021-03-03 22:04:24 -08:00
R Tyler Croy 9d591e64e3 cargo update 2021-03-03 22:04:24 -08:00
R Tyler Croy 504d498846
Merge pull request #62 from nakulpathak3/multiline-comment-grammar 
Add multi-line comment support to parser
 2020-12-12 10:06:16 -08:00
Nakul Pathak 86b7f7d6f4 Add multi-line comment support to parser 
Copied this straight from Pest itself so must be right 😅
https://github.com/pest-parser/pest/pull/332/files

Tested against
```
/* 1-line multiline comment */

/*
  N-line multiline comment
*/

/*
  // Line comment inside multiline
  /*
    (Multiline inside) multiline
  */

*/
```

Testing steps -
* `cargo build`
* `make run`
* `scripts/local-run` with above inputs

Fixes #57
 2020-12-12 01:41:22 -05:00
R Tyler Croy 9011cc0c30 Add a little bit more development documentation to the root readme. 2020-11-29 09:39:02 -08:00
R Tyler Croy b080a10fdd Add a couple more readmes into the crates/ 2020-11-29 09:30:15 -08:00
R Tyler Croy 0b4821137d Implement the parser support for the parallel block 
This was pretty simple, yay.

pipeline {
    parallel {
        stage {
            name = 'Foo'
            steps {
                sh 'pwd'
            }
        }
        stage {
            name = 'Foo'
            steps {
                sh 'pwd'
            }
        }
    }
}
 2020-11-28 22:04:50 -08:00
R Tyler Croy 127d3b7aa4 Refactor the parser code into its own crate for future development 
This will inform #46
 2020-11-28 21:42:48 -08:00
R Tyler Croy 9a39ea00d6 Relocate the agent control socket to outside the current working directory 
There was a race condition where sometimes the agent would set up the control
port so fast that a git into `.` would fail because the directory wasn't empty.

The better solution is to put the control socket outside the workspace in a temp
directory
 2020-11-28 14:46:43 -08:00
R Tyler Croy 8b58f69b5e cargo update 2020-11-28 14:38:47 -08:00
R Tyler Croy 07f1dadea2 Ensure the agent exits with the right status to propagate failures up 
Fixes #51
 2020-11-28 14:38:26 -08:00
R Tyler Croy 02c34b30fd Allow the local-orchestrator to stream agent output into the logs 
This is now the third place I have this same exact code. This should be
refactored into an otto-exec crate and re-used between `sh`, `otto-agent`, and
the local orchestrator
 2020-11-27 21:26:21 -08:00
R Tyler Croy 5664cc1aa2 Make help the default Makefile target 2020-11-27 21:12:21 -08:00
R Tyler Croy f5e4ddbc81 Ensure that stdout and stderr are output together from the sh step 
Fixes #43
 2020-11-27 21:12:20 -08:00
R Tyler Croy 6b3781b746 Add an Ottofile which runs this repos tests 
Basically this requires:

* `make run` to launch the service mesh
* `./scripts/local-run`

And then you wait, because log output from the agent's step invocation is not
yet streaming 😆
 2020-11-27 11:57:05 -08:00
R Tyler Croy d054fdf7ea Correct the parsing of keyword arguments and multiple positional arguments. 
This is basically me stumbling over a shortcut that I left in the parser code
when I first started on it. Now that I am integration testing, I tripped right
into this.

Fixes #44
 2020-11-27 11:47:58 -08:00
R Tyler Croy 74ee25aa1a Ensure that the cache directory for the agent is always absolute 
This ensures that caches are not mistakenly referred to within the pipeline work
directory
 2020-11-27 11:47:21 -08:00
R Tyler Croy 2adb756cbc Start adding some docs for writing steps, only cache covered now 2020-11-26 21:22:25 -08:00
R Tyler Croy 532983f8e6 cargo update 2020-11-26 21:19:51 -08:00
R Tyler Croy dfe6cf9376 Default to caching git repositories with a `git clone --mirror` like functionality 
I first set this up without the "--mirror" like functionality, but unfortunately
I couldn't get my fetch updates to work properly. A future optimization would be
to remove the mirror-refspecs and instead update the refsspecs whenever a new
branch is fetched.

Fixes #40
 2020-11-26 21:18:31 -08:00
R Tyler Croy 7bf5d5fb8f Example what the different directories are about in the main readme 2020-11-26 10:51:09 -08:00
R Tyler Croy c698667c4c Add a couple quick READMEs for the subdirectories 2020-11-26 10:47:52 -08:00
R Tyler Croy ee2510ed6f Refactor the source tree to separate local executables (CLIs) and models a bit more 
This should make the compile cycles a little bit more sensible when just working
on the CLI components of the agent, for example.
 2020-11-26 10:40:37 -08:00
R Tyler Croy 8b9674af3e Ensure the agent is always converting positional parameters to keyword arguments 
Steps are always expecting keyword arguments, but for developer convenience they
can use positional arguments.

This commit properly converts those positional arguments.

Run with my local integration test:

    #!/usr/bin/env ruby

    require 'json'
    require 'net/http'

    pipeline = nil

    # Parse it
    Net::HTTP.start('localhost', 7672) do |http|
        otto = File.read('Ottofile')
        response = http.post('/v1/parse', otto)

        if response.code.to_i != 200
            puts 'Failed to parse file'
            exit 1
        end

        pipeline = JSON.parse(response.read_body)
    end

    # Hit the local-orchestrator
    Net::HTTP.start('localhost', 7673) do |http|
        contexts = []
        pipeline['batches'].each do |batch|
            contexts += (batch['contexts'])
        end

        payload = JSON.dump({
            :pipeline => pipeline['uuid'],
            :contexts => contexts,
        })

        puts payload
        res = http.post('/v1/run', payload)

        if res.code.to_i != 200
            puts "Failed to orchestrate! #{res.code} #{res.read_body}"
            exit 1
        end
        puts 'Enqueued'
    end

Fixes #42
 2020-11-25 21:24:55 -08:00
R Tyler Croy 97088dff12 Run the local-orchestrator inside the tmp directory 2020-11-25 21:18:41 -08:00
R Tyler Croy 308c516e47 Run sequential contexts in the local-orchestrator if there was a success 2020-11-22 20:02:00 -08:00
R Tyler Croy bc8e6e7cec Very simple and naive agent launching in the local orchestrator 
Nothing here is suitable yet, but we're getting there.

Run with:

curl -XPOST -d @test-runworkload.json http://localhost:7673/v1/run

The JSON file:

{
    "pipeline": "2265b5d0-1f70-46de-bf50-f1050e9fac9a",
    "contexts" : [
        {
            "uuid" :"3ce1f6fb-79ca-4564-a47e-98265f53ef7f",
            "properties" : {},
            "steps" : [
                {
                    "symbol":  "sh",
                    "uuid": "5599cffb-f23a-4e0f-a0b9-f74654641b2b",
                    "context": "3ce1f6fb-79ca-4564-a47e-98265f53ef7f",
                    "parameters" : {
                        "script" : "pwd"
                    }
                }
            ]
        }
    ]
}
 2020-11-22 15:50:34 -08:00
R Tyler Croy e21f112ed6 Add some help text to the makefile and introduce the `run` target 2020-11-22 10:47:13 -08:00
R Tyler Croy c7d50516ac Make the parser failure message a bit more compact 
While running fuzzing this was spewing way too many log lines
 2020-11-22 10:46:52 -08:00
R Tyler Croy 6ccb9d841d Sketch out the RunWorkload for the local-orchestrator service 2020-11-22 10:39:47 -08:00
R Tyler Croy 96227d96d5 cargo update 2020-11-21 15:44:45 -08:00
R Tyler Croy 391894ded7 Add the local orchestrator apispecs and a simple healthcheck 
Still not sure what this API should look like, but will start playing around
with it.
 2020-11-21 15:42:10 -08:00
R Tyler Croy e6432839d8 Add a Makefile target for running the apispecs 2020-11-21 15:40:56 -08:00
R Tyler Croy 6db9ca6d75 Update the parser apispec to include the new response format 
Missed this earlier since CI doesn't _yet_ run schemathesis
 2020-11-21 15:04:26 -08:00
R Tyler Croy 508346498d Add the stub for the local-orchestrator 
See #41
 2020-11-21 14:59:27 -08:00
R Tyler Croy b2011e85e1 Fix some shebangs to make the beasties happy 2020-11-20 19:57:31 -08:00
R Tyler Croy 92d7657414 Try running the shunit2 tests for the git step with bash 
In GitHub Actions the tests using pushd/popd are failing which makes me wonder
/bin/sh in those environments is dash or something definitely not bash
 2020-11-20 10:12:04 -08:00
R Tyler Croy cf53ae42dd Start working on providing steps with per-step cache that they can utilize 
This is going to first be used with the `git` step to ensure that it can cache
clones between pipeline runs without hitting the network

See #40
 2020-11-19 20:59:50 -08:00
R Tyler Croy 80425ec62b Give the git step the optional `into` parameter to clone into a specific directory 2020-11-19 20:31:25 -08:00