If one were to have a choice, I would steer clear of jenkins for home use, and use a gitlab runner instead
Jenkins is fine, if you commit to maintaining it, but thats a non trivial task.
THe joy and pain of the gitlab runner is that the job is tied directly to the git repo its attached to. This means that unlike jenkins, the setup is versioned, meaning disaster recovery is much more simple (spin up new runner, attach to project. Done.)
You do then sacrifice multi-project pipelines, but even in very large companies, I've only missed that feature once. It also in no way makes up for having to look after 60+ jenkins masters. (dont ask)
Yes, you gitlab gives you less control over environment, but I would put forward that this is only an advantage when running on windows.
Second this... As someone who's a very small start-up the GitLab CI/CD pipelines are SO much more simple/manageable vs. anything I've ever done with Jenkins.
I self-host GitLab on Linode with a single runner in the cloud (and sometimes use runners on my local machines to speed processes up).
With 15-ish hours of configuration I have my GitLab hosting my SCM, my CI/CD, all of my static HTML sites, my artifact storage, my Docker registry, and even my documentation (I've almost switched 100% over to using their Web IDE to edit markdown).
I've been running this way for over a year and it's incredibly easy to keep maintained. GitLab's upgrade cycles are some of the smoothest around and are always packed with useful features. Overall I've been self-hosting GitLab for about 7 years - it's one of the best software products I've ever used.
Although I love to shill for them - I have no affiliation with GitLab outside of being an enthusiastic user.
Similar situation for me. But I kept using gitlab.com (I didn't want to maintain anything more than I have to)
At $work-1 I had been managing ~800 github repos with circleCI integrations. I was reluctant to stay on gitlab, especially with the hilarious down times in 2018. Two things kept me on there:
1) the runners
2) the ease of organising repos into projects
Now that the SRE team are up to scratch, I don't think I'd go back to github.
If you just want to play with Jenkins, I recommend just getting it up and running in Docker and using only Jenkins Pipelines for jobs and JCasC for system configuration.
This directory (https://github.com/peterwwillis/devops-tools/blob/master/os/...) has basically everything you need to get the manager up and running as a container. (though I think I still need to add Docker to the master container, so you can use the host's Docker daemon to run arbitrary build agent containers on the fly)
From there you can play with the JCasC configuration to do things like add agents, configure the security realm, seed jobs at start-up, etc. Documentation is pretty sparse, but you can at least log in to the web UI and start playing around.
Outside of a single build node, I highly, highly recommend not using Jenkins. It's just such an unnecessarily complicated beast to manage with modern best practices, you might as well just cobble together your own system, or use a real modern distributed build system. It will take you just as long as going through all the endless bullshit options of trying to force Jenkins to do what you want, but you'll actually have something simple and useful in the end that works (if you choose to).
> Outside of a single build node, I highly, highly recommend not using Jenkins. It's just such an unnecessarily complicated beast to manage with modern best practices, you might as well just cobble together your own system, or use a real modern distributed build system. It will take you just as long as going through all the endless bullshit options of trying to force Jenkins to do what you want, but you'll actually have something simple and useful in the end that works (if you choose to).
So much this. I've worked with Jenkins for most of my professional career. Its much better today than what it used to be, but compared to the CI toolings that are now available, using it today makes no sense at all (with a few exceptions, especially when it comes to maven build pipelines).
Regarding suggestions for alternatives to Jenkins I’m curious if anyone has any for game dev use cases.
We use Jenkins now and build for PCs, consoles, and mobile. Almost all of our builds must be on Windows. Some builds require GPUs. Builds require things like specific versions of Unity, Unreal, or a console SDK. Many builds result in build artifacts in the multiple gigabytes which we then want to efficiently copy to e.g. a QA person’s console dev kit on our local network.
For all these reasons we’ve setup our own Jenkins deployment onsite at our office on bare metal. But I’m curious if any of the newer services or technologies might be useful for us.
We're a small startup building infrastructure for developers. We do address problems like what you describe: enforcement of the exact library version requirements for each codebase, reduction of artifact egress costs, integration into the Jenkins workflow as well as into IDE for devs and QA, and more.
We've been looking for game devs who'd be interested in what we're doing.
Please reach out if you're interested.
Thank you for this guide! I am going to have to check it out. I have heard a lot about Jenkins but never really got around to trying it out myself. This seems like a great way to get my feet wet!
> Drone.io does not let developers configure two different projects against the same repository. Instead, one must fork that repository into a new one and use that to create a new Drone.io project.
Very unfortunately, it's the de-facto tool most businesses use. It's free, but you have the option of support, and everyone has used it. There are better free tools, but they're more complicated for the layperson, and usually don't have support, much less the huge ecosystem of plugins.
As a person who earns probably half his living propping up horrible Jenkins installs, I highly recommend that nobody ever use it. But if you're going to work in corporate-world, you probably will have to use it, so you might as well get comfortable with it.
What would you recommend for a shop that has some mature projects they'd like to add a bunch of tests to, to improve speed and confidence in future changes?
Use a SaaS solution like CircleCI, Travis, BuildKite, etc. They all have various upsides and downsides but you will save lots of money vs maintaining you own Jenkins and dealing with all its flaws.
Also, people who are not used to the whole test driven development culture will have an easier time adapting if infrastructure doesn’t constantly break.
Yes, 100%, SaaS is the way to go - unless it is impossible. There are some orgs & projects out there that literally cannot use a SaaS, which usually leads to self-hosting. ("Cost" is not a valid reason, btw - if one more junior manager tells me "oh but it's so much cheaper to self-host" i'm going to take away their lunch money because isn't growing your own lettuce cheaper than paying for it on a sandwich?).
I would say the simplest thing would be to buy a self-hosted GitHub Enterprise, although the cheaper option is self-hosted GitLab CI (but, ugh, maintenance). GitHub Enterprise is so much easier I would gladly pay out the nose for it, though it lags in features.
Next you're looking at Bamboo, TeamCity, Harness, JFrog Pipelines, Spinnaker. Spinnaker is probably the best (only?) open source choice among these.
After that you're looking at open-source projects that come "batteries not included", requiring scripting and integrating more stuff. At that point, my personal preference is to just tie together some AWS or GCloud services, and make your own event-driven task-running pipelines (ex: GitHub webhook triggers Lambda which adds some build items to an SQS queue which triggers more Lambdas which run a build task on Fargate which dumps an artifact into S3 and the end of the pipeline triggers a container build which pulls from S3 artifacts and dumps the resulting container into ECR and test-runs it on Fargate, publishing build results to GitHub commit status). This model is slightly more complex to set up, but insanely easy to scale.
Mainly it's just that the plugins are buggy and undocumented, and everything requires a massive amount of intoning of unnecessary complexity before you figure out how to make it work the way you want, often manually. Its biggest failure is it's not designed to do things in an automated fashion, and it's overkill for what most people actually need to do: run some build scripts when someone commits something.
Outdated? Probably. Its containerized though so running it is much simpler now. The simplicity in running it makes it easier to learn. Learning a tool that is still widely used in the industry is never a bad idea. That's why some people still take the time to learn COBOL.
Jenkins is fine, if you commit to maintaining it, but thats a non trivial task.
THe joy and pain of the gitlab runner is that the job is tied directly to the git repo its attached to. This means that unlike jenkins, the setup is versioned, meaning disaster recovery is much more simple (spin up new runner, attach to project. Done.)
You do then sacrifice multi-project pipelines, but even in very large companies, I've only missed that feature once. It also in no way makes up for having to look after 60+ jenkins masters. (dont ask)
Yes, you gitlab gives you less control over environment, but I would put forward that this is only an advantage when running on windows.