From 921970c8c25383c50166b43d0d25939609e513f4 Mon Sep 17 00:00:00 2001 From: "R. Tyler Croy" Date: Sat, 7 Nov 2020 14:42:31 -0800 Subject: [PATCH] Add some meeting notes from this morning --- ...2020-11-07-pipeline-syntax-discussion.adoc | 124 ++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 meeting-notes/2020-11/2020-11-07-pipeline-syntax-discussion.adoc diff --git a/meeting-notes/2020-11/2020-11-07-pipeline-syntax-discussion.adoc b/meeting-notes/2020-11/2020-11-07-pipeline-syntax-discussion.adoc new file mode 100644 index 0000000..9f0d303 --- /dev/null +++ b/meeting-notes/2020-11/2020-11-07-pipeline-syntax-discussion.adoc @@ -0,0 +1,124 @@ += Pipeline Syntax Discussion + +== Overview + +The goal of this discussion was to share some ideas that +link:https://brokenco.de/2020/11/06/pipeline-syntax-for-otto.html[I had shared] +about Otto Pipeline's syntax with a few colleagues from the Jenkins community. +Both Steven and Gavin have a fair bit of experience in the CI/CD space and +can provide valuable for Otto development at this early stage. + +== Attendees + +* link:https://github.com/rtyler[R Tyler Croy] +* link:https://github.com/steven-terrana[Steven Terrana] +* link:https://github.com/halkeye[Gavin Mogan] + +== Notes + + +* Jenkins Template Engine +** covers the use-case similar to standardBuild, etc Defines the tool agnostic process, templating on the structure of the pipelines (generally). +** Developers can pull their own tools into the templated pipeline +** Uses hierarchy of folders to provide a hierarchy of pipeline structure +** Separate the business logic, the flow of a pipeline, from the step implementations themselves +** All the groovy stuff is in the plugin +** Steps are in Scripted Pipeline +** ~99% of work is configuration parsing, spin up containers, then run stuff +** JT doesn't prescribe any behavior for consumers of templates +* Deploy related discussion +** Previous employeers had a different pipeline for deploy +** Gavin prefers one +** Troubles with post-deploy steps +** Ideally one instance of Jenkins if not one job +** More open trust model personally, doesn't have a problem exposing credentials to a pipeline that can "do it all" +** Steven: +*** JTE exists because couldn't be prescriptive about deployment/etc standards +*** Primarily microservice applications +*** Developers don't get access to pipeline, Jenkinsfile doesn't live in their repo +*** Jenkins is read-only for developers +*** JTE stuff lives external to app repos +* Template versioning +** Folder properties sets the "governance tier" which points to the configuration +** Admins can set different versions of templates, tag pinning. +** Users in JTE can technically BYOT +* Jenkins plugin ecosystem is a reasonable means of sharing re-use +** Centralizing some tooling is beneficial from business continuity +** Upstream helm charts are a mess of if statements to try to do everything +* Declarative pipelines want to just import some specific subsection of a pipeline +** Trying to extend declarative pipelines with shared libraries is perilous +** Re-use has the trouble of relying on others to maintain stuff. +** "Ecosystem is hard" +** Golang's reliance on "just use master" for versioning bit them hard +** Sharing helm charts/YAML is really difficult because of the variance of use-cases +* Benefits to having some opinions to avoid unending complexity +* Step libraries to allow pull-from-everywhere plus "blessed community" +** Setting administration level guards to say "these are blessed sources of steps" can be beneficial a la Shared Libraries +** Marketplace has some connotations of trust relationship with the user's adopting things. +** Install counts matter as a proxy for quality/value of the package +** Digital Ocean's marketplace is curated, but the quality is not terribly high. "Met the requirement at submission time" might be out of date when you look it. +** "17 versions of a `sh` step, which is the one you pick? go by install count" +* Gates and pipeline flow control +** Steven tries to avoid gates in a single pipeline +*** Rather than manual gates, using Git-based flow to drive the pipeline +*** Event-driven is important +** Gavin used BuildKite at a previous job +*** Had a "confirm" step, didn't occupy any resources +** Start a new job to deploy +** Previous place used some orchestration through rundeck external to Jenkins pipelines. +*** Wanted to lock down who could deploy, provided a barrier that made developers don't want to use that. +* Interact with the CLI tool when possible +** Reporting plugins are where plugins really shine +** Worth checking out https://github.com/backstage/backstage +** Home Assistant is a good example of separation of concerns between the frontend/backend line +* What about sharing complex data between re-usable components within the same pipeline +** JTE is using returned data from different parts of the pipeline which can be later accessed. +** "How does the sonarqube step know what tests have been executed?" +* Scale out and fan out/fan-in +** Sauce Labs would do parallelism at the test level +** Jenkins is a novel use-case where we have more matrix +** Deploy to an environment, and then parallel run different types of tests against the staging environment +** depends_on from Terraform could be interesting to import +** JTE has "lifecycle hooks" for beforestep/afterstep/cleanup/etc, which instructs the engine to run things for you automatically. +** aspect oriented or graph syntax +** Argo sync waves are an interesting model to look at +* Different names for hte behaviors rather than a bunch of options/configurations +* Is stage an optional construct? +** Purely aesthetics for wrapping steps +** Largely to make things look better in the UI in the case of Scripted Pipeline + + +.Exploring syntax for more "root" verbs +[source] +---- +pipeline { + stage { + name = 'Deploy to Stage' + steps { } + } + fanout { + stage { + name = 'User Acceptance Tests' + } + stage { + name = 'Accessibility Tests' + } + } + wait { + // describe a gate type event + } + pause { + } + parallel { + } +} +---- + +In the above example, rather than trying to model nested stages or different +incantations of stage-level options, what if Otto just had more verbs at the +"root" level to define the workflow? The benefit of this approach is that +`wait` or `pause` can be specifically put external to stages and therefore +agent allocation, making it a lot more clear that there is some event the +execution is waiting for. + +