Buildx also has a completely different build output and it's completely useless for anything. Use `--output=plain`.
Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around.
I've had cases where BuildKit will get stuck and `docker buildx prune -a` will leave things around. The way to fix it is with `systemctl restart containerd.io`. This seems to happen a lot when I interrupt the build (eg, I know it will fail and need to restart it with a fix), but it's also happened on successful builds.
All of that is on top of docker itself having extremely poor error messages.
If you build your dockerfile with earthly, we have a -i flag that will pop you into a shell on a failure, and we are using buildkit, so it's totally possible.
> Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around.
We ended up switching to using multiarch/qemu-user-static:7.2.0-1 <https://hub.docker.com/layers/multiarch/qemu-user-static/7.2...> which also mutates binfmt_misc in buildx's context in order to exec the static copy of qemu in it but I find their shell script much more legible about what's going on: <https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/...> not to mention the fact that it more clearly represents the recent qemu version versus secreting it away inside the moby/buildkit:heheh-good-luck-friend
I see you're trying to do multi-arch builds. At reasonable (on bare metal) speeds you say?
Delete the entire docker stack. It's utter shite. Then install Podman and Buildah, and rejoice! For all the evil buildx magic can now be done explicitly.
This solution was born when trying to do native builds on CircleCI - where you do not have a buildx cluster with multiple architectures. You have to build the archs as independent build steps, then pull them together afterwards. This is something that Docker never considered, and Docker only functions on the happy path. Buildah has multiarch manipulation commands: https://danmanners.com/posts/2022-01-buildah-multi-arch/ (not a plug, first blog on search)
We migrated to GHA, but still use separate builders to avoid the Docker trite. We also ditched DCT (Docker signing) for Cosign because - again - DCT is home-grown garbage.
Similar to sibling comment, mostly just works out of the box. Things I've noticed that are different:
- There's no docker.sock API socket by default, which makes sense because Podman doesn't have Docker's daemon-based architecture. You can run one with `podman system service [...]` if you have non-CLI clients expecting to connect to some $DOCKER_HOST
- Mounts/volumes behave subtly differently IIRC, I had a bunch of scripts that didn't work with podman-as-docker out of the box, I think the issue was that Docker will create some directories automatically that Podman won't?
Aside from that, rootless Podman is great and I've had far fewer issues with it than I did for rootless Docker, only real caveat is that -p <some port> doesn't actually connect to the host, it's still namespaced. You have to use --net=host for actual host networking.
I recently took the plunge in a couple of my CI pipelines and had little trouble. The only thing that actually broke was that the `--cache-from` option doesn't accept a tag, where it does for docker. Everything else pretty much Just Worked.
That said, my pipelines are fairly simple, as are my Dockerfiles, so...
It shaved significant time off my pipelines, though, not having to wait for a docker-in-docker service to spin up (I use Gitlab with Kubernetes runners).
I've had a good experience with Podman as a Docker alternative. The Podman CLI is designed to be API-compatible with Docker's CLI, so most things should just work. The only thing I've had to change with my Dockerfiles (or Containerfiles) is the `SHELL` instruction, since Podman by default builds OCI images, and that instruction is not supported by the OCI standard. But you can still specify the shell individually for the `RUN` instructions, or even just use `--format=docker` if you don't want OCI images.
I've spent a bunch of time playing around with Docker's API directly. BuildKit is notable because its API is completely undocumented AFAICT - certainly the API listed under https://docs.docker.com/engine/api/v1.44/#tag/Image/operatio... is just the API for a normal non-BuildKit build. If you want a BuildKit build, you have to use the Docker CLI.
I did some digging and wiresharking, and I'm pretty sure it's undocumented because the API is _insane_. It starts as HTTP/1 from the Docker client to the Docker engine, but a key BuildKit feature is that the engine pulls files & data from the client on-demand, which is hard in a normal REST API, so how does it do that? By renegotiating the connection to flip the direction after the client connects.
That means: the client sends an HTTP/1 request, the server offers to upgrade to HTTP/2 by in reverse, and then the server becomes an HTTP client and the client becomes an HTTP server, still on the same existing connection. All actual communication then happens as gRPC, but backwards.
This is one of the things that led us to write this post! It's almost all undocumented, as you're mentioning, and there are quite a lot of complexities underneath the surface that often feel very inefficient as well.
We're working on another project in this realm that might interest you. Happy to send over more details via email if your interested in better build APIs. My contact info is in my profile.
Does anyone have a good reference on how exactly Docker/BuildKit/Podman/… build Dockerfiles or, more generally, run containers using the underlying OCI container runtime (runc/crun/…)? I think I have a rough understanding but I'm particularly interested in how exactly the rootless setup works, the details of layer caching (beyond "it uses overlayfs"), etc.
Thanks, I had come across that post already but I find the structure of OCI images to be the easy part to understand. The tougher part is to get runc to run everything without root. :)
Anyway, fortunately I seem to have found a solution for now (running runc with an overlay rootfs without root), see the link in the other sibling/nephew comment I posted.
> I got familiar with container intervals via, basically, "building images without docker"
That's been my approach, as well. However, right now I'm stuck at getting overlayfs to work without privileges (easy: use `unshare`), while not breaking rootless runc (apparently not so easy).
Did you ever try running any containers without pushing to a registry? That is the part I can't wrap my head around yet? How do I get the run time to start the container locally?
You can build an image with docker, and run it locally.
You can also use docker save to get a tarball and ship that file to another machine, which can be run through docker load and then run as if it was built locally.
Hey, thanks for the question! Depot co-founder here.
We've optimized BuildKit for remote container builds with Depot. So we've added things like a different `--load` for pulling the image back that is optimized to only pull the layers back that have actually changed between the build and what is on the client. We've also done things like automatically supporting eStargz, adding the ability to `--push` and `--load` at the same time, and the ability to push to multiple registries in parallel.
We've removed saving/loading layer cache over the network. Instead, the BuildKit builder is ephemeral, and we orchestrate the cache across builds by persisting the layer cache to Ceph and reattaching it on the next build.
The largest speedup with Depot is that we build on native CPUs so that you can avoid emulation. We run native Intel and Arm builders with 16 CPUs and 32GB of memory inside of AWS. We also have the ability to run these builders in your own cloud account with a self-hosted data plane.
So the bulk of the speed comes from persisting layer cache across builds with Ceph and native CPUs. The optimized portions of BuildKit really help post-build currently. That said, we are working on some things in the middle of the build related to the DAG structure of BuildKit that will also optimize up in front of the build.
Seeing that reminded me of some healthy discussion in https://news.ycombinator.com/item?id=39235593(SeaweedFS fast distributed storage system for blobs, objects, files and datalake) that may interest you. control-f for "Ceph" to see why it caught my eye
hopefully depot will reply, but from my perspective it is mostly laid out on their homepage. they are comparing against builds in other CI products that use network-backed disks, virtualized hardware, and don’t keep a layer cache around. Depot provides fast hardware and disks and is good at making the layer cache available for subsequent builds.
You could likely get very similar performance by provisioning a single host with good hardware and simply leverage the on-host cache.
For instance, building up the DAG of all the build steps and then scheduling things so that various parts can be built in parallel. Buildkit does a lot that we use beyond just being a way do build things inside a runc container.
That said, we have a fork of buildkit for the various things we add that don't fit well in the upstream.
Already our auto-skip feature and our branching are implemented on top of Buildkit rather than using it. Probably as we grow and add more build centric features we will continue to diverge.
I'm just a DevRel person though, so that's just my 2 cents. The core team may disagree with me.
I am considering writing a small library / framework to optimize scheduled tasks by making a DAG and inferring which ones are safe to be parallel and which aren't. So this can be useful.
Dagger (https://dagger.io) is a great way to use BuildKit through language SDKs. It's such a better paradigm than Dockerfiles, I cannot imagine going back.
Dagger is by the same folks that brought us Docker. This is their fresh take on solving the problem of container building and much more. BuildKit can more than build images and Dagger unlocks it for you.
Dagger is quite cool, and we've been doing some collaboration with their new Zenith stuff. We actually have a module where you can route an image build in your Dagger pipeline to Depot and do interesting things with the result as we stage your image in an ephemeral registry [0].
Caching is definitely their product and one of the main benefits.
1. You can get the developer UX improvements without shared caching
2. You can run it in k8s and get shared caches there. I believe it should support any BuildKit caching solutions, and they will help you figure that out in Discord
Staged docker builds seem like a bad idea. Why would you want to tie your build specification with your runtime specification which are inherently different concerns? You lose out on independent evolution and isolation between the two. And for organizations, it forces a structure where a single team needs to manage both.
What problems does it solve that cannot be easily achieved with separate build and runtime files?
I use both, so here is what I'm doing and why I find value in staged Docker builds.
I am running a (relatively new) NX monorepo, which deploys ~6 microservices to various K8s environments using Skaffold. All services are Typescript, Node 20.11, with pnpm as a package manager.
I have separate build files for certain common base images, which mostly are Node-($v)-alpine with a few CI/common dependencies installed. I use Skaffold to compose the image builds.
When I Dockerize each service, I need to install dependencies for that specific service only, and in that case the install/build specification must be the same as the runtime specification. Rather than creating separate build image files and composing them with Skaffold, I simply use a build stage to install the microservice dependencies (using one of the base images with NX and PNPM), then copy them to the final image (which does not have those deps). This is a pretty common pattern, and I like working with it. The advantages to build stages are readability and simplicity (no extra orchestration needed), and the whole setup makes it very difficult to make mistakes.
Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around.
I've had cases where BuildKit will get stuck and `docker buildx prune -a` will leave things around. The way to fix it is with `systemctl restart containerd.io`. This seems to happen a lot when I interrupt the build (eg, I know it will fail and need to restart it with a fix), but it's also happened on successful builds.
All of that is on top of docker itself having extremely poor error messages.
I really do not like docker or buildx.