R Tyler Croy
f8dc3af95e
Update the cargo versions
...
Fixes #64
2021-03-03 22:04:25 -08:00
R Tyler Croy
102ded9756
Finally fiddled around enough to make this work
2021-03-03 22:04:25 -08:00
R Tyler Croy
2ebf41bb31
This is just awful
...
Hopefully finding better ways in https://github.com/launchbadge/sqlx/issues/1083
2021-03-03 22:04:25 -08:00
R Tyler Croy
14d78e009f
Check pointing some exploration with graphql for the reldata service
2021-03-03 22:04:25 -08:00
R Tyler Croy
08d584974b
Upgrade to tide 0.16 across the board to make the async-graphql example code work
2021-03-03 22:04:25 -08:00
R Tyler Croy
78534ca7d1
Add some Makefile targets to make working with sqlx a little more simple
2021-03-03 22:04:25 -08:00
R Tyler Croy
8a1a47669c
Scaffolding up reldata service's data migrations
...
Right now I'm just working within sqlite, but the migrations will need to be
mirrored for PostgreSQL in the future in `migrations/postgres`
Running this entails using the sqlx-cli (`cargo install sqlx-cli`)
% sqlx migrate --source migrations/sqlite run
2021-03-03 22:04:25 -08:00
R Tyler Croy
f017ea4660
Pull in the async-graphql dependency for reldata
...
I'm hoping that I can map objects through with graphql relatively easily so I
don't have to go full REST API for this service
See #63
2021-03-03 22:04:25 -08:00
R Tyler Croy
4e9c851dd6
Refactor some dependencies to allow jumping to async-std 1.9.0
...
There are some good stabilizations and other work in this release that I want.
Also, my Cargo.tomls have been too strict, need to start relaxing that
2021-03-03 22:04:25 -08:00
R Tyler Croy
60e0b82247
Add a stub relational data service while I explore GraphQL
...
See #63
2021-03-03 22:04:25 -08:00
R Tyler Croy
a9adeb2d8a
Map the project and system configuration defined in the demo directory
2021-03-03 22:04:24 -08:00
R Tyler Croy
0412bf4226
Sketching out some system and service specific configuration
...
The `demo/config` directory is eventually what I would expect to live in
`/etc/otto` on Otto hosts.
See #39 and #52
2021-03-03 22:04:24 -08:00
R Tyler Croy
9d591e64e3
cargo update
2021-03-03 22:04:24 -08:00
R Tyler Croy
504d498846
Merge pull request #62 from nakulpathak3/multiline-comment-grammar
...
Add multi-line comment support to parser
2020-12-12 10:06:16 -08:00
Nakul Pathak
86b7f7d6f4
Add multi-line comment support to parser
...
Copied this straight from Pest itself so must be right 😅
https://github.com/pest-parser/pest/pull/332/files
Tested against
```
/* 1-line multiline comment */
/*
N-line multiline comment
*/
/*
// Line comment inside multiline
/*
(Multiline inside) multiline
*/
*/
```
Testing steps -
* `cargo build`
* `make run`
* `scripts/local-run` with above inputs
Fixes #57
2020-12-12 01:41:22 -05:00
R Tyler Croy
9011cc0c30
Add a little bit more development documentation to the root readme.
2020-11-29 09:39:02 -08:00
R Tyler Croy
b080a10fdd
Add a couple more readmes into the crates/
2020-11-29 09:30:15 -08:00
R Tyler Croy
0b4821137d
Implement the parser support for the parallel block
...
This was pretty simple, yay.
pipeline {
parallel {
stage {
name = 'Foo'
steps {
sh 'pwd'
}
}
stage {
name = 'Foo'
steps {
sh 'pwd'
}
}
}
}
2020-11-28 22:04:50 -08:00
R Tyler Croy
127d3b7aa4
Refactor the parser code into its own crate for future development
...
This will inform #46
2020-11-28 21:42:48 -08:00
R Tyler Croy
9a39ea00d6
Relocate the agent control socket to outside the current working directory
...
There was a race condition where sometimes the agent would set up the control
port so fast that a git into `.` would fail because the directory wasn't empty.
The better solution is to put the control socket outside the workspace in a temp
directory
2020-11-28 14:46:43 -08:00
R Tyler Croy
8b58f69b5e
cargo update
2020-11-28 14:38:47 -08:00
R Tyler Croy
07f1dadea2
Ensure the agent exits with the right status to propagate failures up
...
Fixes #51
2020-11-28 14:38:26 -08:00
R Tyler Croy
02c34b30fd
Allow the local-orchestrator to stream agent output into the logs
...
This is now the third place I have this same exact code. This should be
refactored into an otto-exec crate and re-used between `sh`, `otto-agent`, and
the local orchestrator
2020-11-27 21:26:21 -08:00
R Tyler Croy
5664cc1aa2
Make help the default Makefile target
2020-11-27 21:12:21 -08:00
R Tyler Croy
f5e4ddbc81
Ensure that stdout and stderr are output together from the sh step
...
Fixes #43
2020-11-27 21:12:20 -08:00
R Tyler Croy
6b3781b746
Add an Ottofile which runs this repos tests
...
Basically this requires:
* `make run` to launch the service mesh
* `./scripts/local-run`
And then you wait, because log output from the agent's step invocation is not
yet streaming 😆
2020-11-27 11:57:05 -08:00
R Tyler Croy
d054fdf7ea
Correct the parsing of keyword arguments and multiple positional arguments.
...
This is basically me stumbling over a shortcut that I left in the parser code
when I first started on it. Now that I am integration testing, I tripped right
into this.
Fixes #44
2020-11-27 11:47:58 -08:00
R Tyler Croy
74ee25aa1a
Ensure that the cache directory for the agent is always absolute
...
This ensures that caches are not mistakenly referred to within the pipeline work
directory
2020-11-27 11:47:21 -08:00
R Tyler Croy
2adb756cbc
Start adding some docs for writing steps, only cache covered now
2020-11-26 21:22:25 -08:00
R Tyler Croy
532983f8e6
cargo update
2020-11-26 21:19:51 -08:00
R Tyler Croy
dfe6cf9376
Default to caching git repositories with a `git clone --mirror` like functionality
...
I first set this up without the "--mirror" like functionality, but unfortunately
I couldn't get my fetch updates to work properly. A future optimization would be
to remove the mirror-refspecs and instead update the refsspecs whenever a new
branch is fetched.
Fixes #40
2020-11-26 21:18:31 -08:00
R Tyler Croy
7bf5d5fb8f
Example what the different directories are about in the main readme
2020-11-26 10:51:09 -08:00
R Tyler Croy
c698667c4c
Add a couple quick READMEs for the subdirectories
2020-11-26 10:47:52 -08:00
R Tyler Croy
ee2510ed6f
Refactor the source tree to separate local executables (CLIs) and models a bit more
...
This should make the compile cycles a little bit more sensible when just working
on the CLI components of the agent, for example.
2020-11-26 10:40:37 -08:00
R Tyler Croy
8b9674af3e
Ensure the agent is always converting positional parameters to keyword arguments
...
Steps are always expecting keyword arguments, but for developer convenience they
can use positional arguments.
This commit properly converts those positional arguments.
Run with my local integration test:
#!/usr/bin/env ruby
require 'json'
require 'net/http'
pipeline = nil
# Parse it
Net::HTTP.start('localhost', 7672) do |http|
otto = File.read('Ottofile')
response = http.post('/v1/parse', otto)
if response.code.to_i != 200
puts 'Failed to parse file'
exit 1
end
pipeline = JSON.parse(response.read_body)
end
# Hit the local-orchestrator
Net::HTTP.start('localhost', 7673) do |http|
contexts = []
pipeline['batches'].each do |batch|
contexts += (batch['contexts'])
end
payload = JSON.dump({
:pipeline => pipeline['uuid'],
:contexts => contexts,
})
puts payload
res = http.post('/v1/run', payload)
if res.code.to_i != 200
puts "Failed to orchestrate! #{res.code} #{res.read_body}"
exit 1
end
puts 'Enqueued'
end
Fixes #42
2020-11-25 21:24:55 -08:00
R Tyler Croy
97088dff12
Run the local-orchestrator inside the tmp directory
2020-11-25 21:18:41 -08:00
R Tyler Croy
308c516e47
Run sequential contexts in the local-orchestrator if there was a success
2020-11-22 20:02:00 -08:00
R Tyler Croy
bc8e6e7cec
Very simple and naive agent launching in the local orchestrator
...
Nothing here is suitable yet, but we're getting there.
Run with:
curl -XPOST -d @test-runworkload.json http://localhost:7673/v1/run
The JSON file:
{
"pipeline": "2265b5d0-1f70-46de-bf50-f1050e9fac9a",
"contexts" : [
{
"uuid" :"3ce1f6fb-79ca-4564-a47e-98265f53ef7f",
"properties" : {},
"steps" : [
{
"symbol": "sh",
"uuid": "5599cffb-f23a-4e0f-a0b9-f74654641b2b",
"context": "3ce1f6fb-79ca-4564-a47e-98265f53ef7f",
"parameters" : {
"script" : "pwd"
}
}
]
}
]
}
2020-11-22 15:50:34 -08:00
R Tyler Croy
e21f112ed6
Add some help text to the makefile and introduce the `run` target
2020-11-22 10:47:13 -08:00
R Tyler Croy
c7d50516ac
Make the parser failure message a bit more compact
...
While running fuzzing this was spewing way too many log lines
2020-11-22 10:46:52 -08:00
R Tyler Croy
6ccb9d841d
Sketch out the RunWorkload for the local-orchestrator service
2020-11-22 10:39:47 -08:00
R Tyler Croy
96227d96d5
cargo update
2020-11-21 15:44:45 -08:00
R Tyler Croy
391894ded7
Add the local orchestrator apispecs and a simple healthcheck
...
Still not sure what this API should look like, but will start playing around
with it.
2020-11-21 15:42:10 -08:00
R Tyler Croy
e6432839d8
Add a Makefile target for running the apispecs
2020-11-21 15:40:56 -08:00
R Tyler Croy
6db9ca6d75
Update the parser apispec to include the new response format
...
Missed this earlier since CI doesn't _yet_ run schemathesis
2020-11-21 15:04:26 -08:00
R Tyler Croy
508346498d
Add the stub for the local-orchestrator
...
See #41
2020-11-21 14:59:27 -08:00
R Tyler Croy
b2011e85e1
Fix some shebangs to make the beasties happy
2020-11-20 19:57:31 -08:00
R Tyler Croy
92d7657414
Try running the shunit2 tests for the git step with bash
...
In GitHub Actions the tests using pushd/popd are failing which makes me wonder
/bin/sh in those environments is dash or something definitely not bash
2020-11-20 10:12:04 -08:00
R Tyler Croy
cf53ae42dd
Start working on providing steps with per-step cache that they can utilize
...
This is going to first be used with the `git` step to ensure that it can cache
clones between pipeline runs without hitting the network
See #40
2020-11-19 20:59:50 -08:00
R Tyler Croy
80425ec62b
Give the git step the optional `into` parameter to clone into a specific directory
2020-11-19 20:31:25 -08:00