But as always with “”gitops”” themed tools I think it’s pretty awkward to handle rollbacks. Either you take the stance that master is the source of truth and let the CI/CD tooling revert commits if deployment fails, or you store that state elsewhere and allow e.g manifests to diverge from git.

I would be curious to hear how other people do it

In our case, we use trunk-based development where "main" is the branch that is approved for release, but it does not necessarily point to what is currently released. Instead, we use git tags for that. On merge to main, we kick off pipeline executions (we use AWS CodePipeline here, but Github actions with a concurrency group id set would achieve something similar). Non-prod stages just push lightweight tags in the format of `$STAGE-YYYY.MM.DD.hhmm` on successful deployment. Prod stages are similar, but we publish a Github release declaration using the Github API rather than just pushing a git tag. In the case of rollback, you either push new tags at older commits or roll forward with a revert commit (but still using tags as the tracking mechanism).

I wonder if the tag based release is a widespread industry pattern (especially for growing monorepos)? Would be curious to learn more about it

External releases are built on release branches off of main with their own git tag rules, but I want to touch on internal releases off of main.

Between feature branches and main we have GitHub status checks that gate bad PRs from getting in. Once the PRs are in main, we build a nightly full suite of components and put them through various levels of interop testing. Once our system receives the signal that all builds and interop testing had passed, it applies a Last Known Good (LKG) tag to the git commit the components were built from.

After that, various systems including artifact links for QA and auto-merge jobs from main, are set to use the LKG tag to ensure they are getting good builds and code.

[1] https://github.com/nektos/act

Toast lets you use whatever base image you want (even when running with GitHub Actions), and it has some extra features for local development (e.g., caching, bind mounts, tasks with dependencies between them, etc.).

[1] https://buildjet.com/for-github-actions

edit: I decided to upgrade and give it a try, went from 0.2.21 to 0.2.25 (Mar 30, 2021 → Nov 24, 2021). It's still failing to run a basic "build and test" pipeline, and really not doing anything fancy: apt-get install + make + ./service + ./tests.py, with a Redis sidecar. This seems to be because services are not supported in `act`: https://github.com/nektos/act/issues/173

this was all before I knew about act and it's about the same speed in my limited testing.

One of the best tutorials for people coming to Go from other languages:

https://gobyexample.com/

We shouldn't be writing all these jobs in a format that only works for one CI system. You spend months writing Jenkinsfiles, and then you move to CircleCI and have to rewrite all of them, and then move to Drone and have to rewrite all of them, and then move to CodeBuild/CodePipeline and have to rewrite all of them, and then move to GitHub Actions and have to rewrite all of them. Eventually we'll rewrite them all for something else. And why? To run the same exact tests on a slightly different system.

In general though I try to put all the CI work into a simple shell or similar script and then just configure whatever CI system is in use to call it in the appropriate environment (in a container, bespoke VM, etc.). I agree putting all kinds of complexity in a unique CI system is just asking for trouble down the road as it's now basically a hard dependency for your code to be shippable.

It doesn't? https://github.blog/changelog/2021-11-10-github-actions-inpu...

Maybe it didn't 6 months ago, but it seems this is an option now, and well-documented.

This is a topic of interest for me, I presented on Jenkins and GitOps to an audience at KubeCon who I suspect are almost all interested in moving away from Jenkins, or have been told to be interested in switching from Jenkins to something else, and I tried to get the idea across that they probably don't really need to switch the workflow tool even if it's ancient, ...

But maybe should consider subbing out some of the important fiddly bits underneath it (like, I assume the vast majority of Jenkins users are building images with Docker, and if they're running on Kubernetes, they're many of them wondering what they will do, or how long they really have before they have to start worrying about the deprecation of dockershim and how their lives are going to change when their clusters won't be running Docker under the hood anymore?)

I was actually arguing for a tool like Porter.sh to come in and make the boundaries of "what's in a build" super neat and tidy, organized, but also limited so the next time they feel compelled to switch workflow tooling, it will be a non-issue and can be over and finished inside of a single day's work. The problem isn't that your workflow tool is too old, it's that you've jammed too much arbitrary complexity inside of it, probably because the right abstraction was not made available to you at the time. "Switching off of Jenkins" just means building a second system and it comes with all the baggage of "second system syndrome" to do so.

Sure it's difficult switching from Jenkins, when you've built this gigantic pipeline with 18 branches and 12 of them run in parallel, half of them are configured with different options passed to the docker build tool, half of them must use buildx, and the third half of them are unmaintained so we don't go in there... so put some guard rails up around the hard parts! And get somebody in there to take care of those cobwebs.

I think we need a few more "12 Factor App"-style guidelines for modern systems. 12 Factor goes a long way to abstract away the tendency for implementation lock-in. But we can probably create a few more guidelines specific to CI so it's portable. Use OAuth, use the same authorization layer as your VCS, tie secrets and artifacts to the VCS repo, make every plugin a Docker container. (I'm stealing these concepts from Drone because it has the best cloud-native design I've ever seen)

The final unsolved bit is how to manage a DAG of jobs and hooks around every event in a portable way. We probably need a universal CI spec and API, but maybe that's too specific.

Also: holy crap, I don't think I've seen Porter.sh before, I love the idea! Definitely going to look into that

Someone else found the link.

The original example was that in LISP it's so easy to add Object Oriented functionality that it's a mere afternoon's work for a graduate student. For comparison, it took Bjarne Stroustrup several years to extend C with objects to create C++. Hence there are only about 4-5 such languages out there, of which only two are popular.

This means that every C++ program is object oriented in the "same way", whereas random OO LISP modules are incompatible and you can't merged them into one cohesive program or reuse the components nicely.

It's almost trivial to come up with a build system. Like you said, it boils down to not much more than interpreting a simple sequence of steps, typically doing nothing more than just triggering shell execution directly or indirectly. A basic build system is a few days work for a talented programmer! Heck, a mere shell script will do in a pinch...

Notice however how the industry is slowly consolidating on Docker and Kubernetes. I suspect that one reason for this is that these are hard technologies. Not just anyone can "whip up" a container build and orchestration system in a short period of time.

Hence, there's only a handful of container systems commonly used in the wild, which has resulted in standardisation. Precisely the type of standardisation that you (and many others) have been craving.

TL;DR: The most complex system will become the CI system standard to rule them all because it's complex.

MS moved a lot of (almost all) Azure DevOps people and put them on GitHub.

Saying a lot and causing confusion usually go hand-in-hand.

GHA isn’t a product thought for GH private organisations, you will find that every much needed feature for this use is very low in GH roadmap.

However I'd say that Github's been pretty receptive to feedback and has actively fixed almost every wall that we've run into (if we haven't been able to fix it for ourselves)

It doesn’t matter how smart you are with reusable workflows you will never get to a truly DRY setup that scales for dozens of repositories.

Another major pain is that we still haven’t private actions. It was due end of 2021 (maybe it is out now but I checked a couple of days ago).

Setting up runners to look after a pool of repositories needs elevated permissions.

GH offers a way to enforce a list of enabled actions but this does not work with private binary registries hosting pre built Docker actions. The only thing that could prevent you to pull software at runtime from the internet, which means, if you want to have a decent security posture all you are left with is referencing actions using the full git sha version.

Many common use case require hacks, which is fun for a weekend project but isn’t great for a large scale operation. An example is simply running a workflow dynamically targeting the folders containing changes. At the moment you have to create a job, generate a build matrix on the fly and pass it in input as the matrix to the actual job.

I'm looking forward to this landing too. In the meantime, though, checking out the repository that contains the Actions and referencing a local path works fine so this hasn't really been a blocker for us.

Edit: per sibling comment, it seems that this feature became available in the last few days. Nice!

- Go By Example: https://gobyexample.com/

- Rust By Example: https://doc.rust-lang.org/rust-by-example/

- V [a weird knockoff of Go] By Example: https://v-community.gitbook.io/v-by-example/

There's also 'Learn X in Y Minutes' (https://learnxinyminutes.com/), which covers a range of different 'X'es. They make it ridiculously easy to get going with a new tool/language, IMO. It's a superb paradigm in general.

Aside from the typo, I wonder how many packages could be backdoored at once, if an action maintainer went rogue, seeing as there's no pinning for actions by default, and (according to https://github.com/msys2/setup-msys2/blob/main/HACKING.md) moving a tag is the default way to push updates to an action. (Interestingly get-cmake/run-cmake/run-vcpkg are all operated by the same person.)

If anyone is interested to mitigate it yourself, these are helpful :)

https://docs.github.com/en/actions/creating-actions/about-cu...

https://github.com/dependabot/dependabot-core/issues/2835

https://github.com/zgosalvez/github-actions-ensure-sha-pinne...

https://github.com/timmeinerzhagen/dependabot-sha-comment-ac...

I agree that it’s a bit inelegant though.

Shot in the dark, anyone know of one for hot-off-the-presses modern Python (3.10) with typing akin to Golang? All the modern additions really need a comprehensive overview like Go by Example somehow manages to do in a very lightweight style

Probably worth checking out this guide. GHA can be a pretty scary thing.

My current solution doesn't support spaces within the secret content:

  - name: Write .env
    run: |
      echo $ENV_FILE | tr ' ' '\n' > .env
    shell: bash
    env:
      ENV_FILE: ${{secrets.DOTENV}}

- Reusable workflows (note: matrix strategy doesn't work here): https://docs.github.com/en/actions/using-workflows/reusing-w...

- Creating an action: https://docs.github.com/en/actions/creating-actions/metadata...

- Composite actions: https://docs.github.com/en/actions/creating-actions/creating...

- Fact, the "uses" in step can be a relative path in the same repository (obviously, you must checkout the code, when it uses a relative path): https://docs.github.com/en/actions/using-workflows/workflow-...

- Script as an action: https://github.com/actions/github-script

- Using GitHub Packages and artifacts: https://docs.github.com/en/actions/publishing-packages/about...

- Using docker-compose-like services that run alongside of the container: https://docs.github.com/en/actions/using-containerized-servi...

- Using heredoc to share multi-line JSON in an environment variable, also using fromJSON/toJSON functions:

  - name: get-version
    id: get-version
    run: |-
      echo 'JSON_RESPONSE<<EOF' >> $GITHUB_ENV
      cat package.json >> $GITHUB_ENV
      echo 'EOF' >> $GITHUB_ENV

  - name: print-version
    run: echo "${{ fromJSON(env.JSON_RESPONSE).version }}"

And many, many more :)