Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept. The servers themselves are quite simple (single .jar, etc). These days we have containers, and it's made CI a much better thing.
I agree with this. The quality of your Jenkins experience will be pretty much inversely proportional to how much groovy script is in your Jenkinsfiles.
Every new permutation of parameters is a new chance for something to go wrong, and there's basically no sane way to unit test or validate any of your "helper" functions, so they're all write-once-change-never.
Which is not necessarily a problem inherent in Jenkins— Jenkins just gives you more than enough rope to hang yourself. GitLab CI can get silly too, but culturally it inherits from the much simpler Travis model, where you're basically expected to just be wrapping some other build tool (eg, tox from the Python world) rather than rolling the whole thing yourself.
Groovy is arguably the wrong language choice, even if it worked fine in other aspects. The string handling situation alone is crazy, which is compounded if you are using it with bash (which has its own crazy ideas about string escaping). That's a not too uncommon scenario.
To be fair, basically everything that interops/embeds bash like this ends up in a similar place— you see the same double-escape ridiculousness in CMakelists files and Nix package definitions.
> Jenkins is a nightmare because everyone decided to spend months customising it with groovy and it became overly specific to your team/dept
Yes. One thousand times this.
If you are running simple scripts it's fine. If you are using the declarative pipeline, it's fine. The moment you start adding Groovy you'll be down a path that is filled with sadness and anger. Mind you, even the folks behind Jenkins will advise you not to use any complex Groovy scripts(including for performance reasons - you can easily overwork the jenkins master).
I've been focusing on Concourse because it forces the usage of containers for everything. You don't have to care about what's installed in the worker node, you just use a container that has the stuff you want. Simple inputs and outputs.
You _can_ do the same sort of thing with Jenkins (but be aware of all the bugs still open regarding containers). But Jenkins doesn't force you to do anything, nor it gives you an easy and out of the box solution to string containers together. Left unchecked, you have your reproducible builds running in completely unreproduceable magical build machines – that noone really understands how it all works.
Did a migration of a few hundreds of pipelines from one server to another and it exposed a lot of dependencies we didn't know we had. Plugins, jars, packages installed in build machines, you name it.
If you must use Jenkins, please try to avoid Groovy to the max, write anything that's non-trivial as an executable (even if it is a bash script) and call it from the main pipeline. Use containers if you can to avoid build machine dependencies. Try to use declarative pipelines too unless your jobs are very simple, and avoid the 'script' blocks. Do not use the scripting pipeline to avoid inviting groovy to your home.
I worked on a Jenkins cluster with around 150 agents and many, many builds per day. It all ran so smoothly and I swear it was because there was zero groovy.
