Shameless plug: I wrote undocker[1] to convert docker images to a rootfs tarball, so I can run them with plain systemd.
Goal: no more daemons to run 3rd party containers, systemd is good enough by now: resource limits, isolation, chroot, dynamic users, logging, and more.
Low-level tooling is done, I am now building ecosystem around it: easy installation, convert to deb/rpm, systemd units, etc.
Been using systemd for personal project deployments for quite a while now after leaving ZEIT (Vercel) and the year or so fighting endlessly with k8s and docker's shortcomings.
I love it. It does everything I need it to, it's fast, decently documented, supports everything Docker does (just not 1:1 in terms of DX) and is present wherever I need it.
I really like this. At least for the nspawn case, systemd as of 242 supports loading containers from OCI images directly, but still doesn't seem to support OCI hooks or have a runc-compatible CLI, which were the other two outstanding issues identified by Poettering before it could be considered a true drop-in replacement for running container bundles loaded from OCI images.
As for just using systemd and not nspawn at all, that seems useful for a lot of cases, but not multiplexing of networked services since systemd can't assign UNIX domain hostnames and IP addresses to individual services. At least, I think it can't, can it?
ProtectHostname will generate a new hostname for the service process hierarchy.
JoinsNamespaceOf can be used to set up network namespaces (albeit in a clunky way).
It is possible, but multiplexing networked services requires significantly more effort with systemd than with others. I do not have a strong use case yet to invest in tooling for it, but it may come.
Your goals seem to somewhat align with the podman project, except that they decided to make a drop in replacement for docker (to the point you can alias docker to podman) and leverage the docker ecosystem, and still use orchestration to build pods, etc. Notably, it's no longer a daemon, just some python scripts that do the same thing docker did.
Also, as good as systemd is (and I've defended it here a few times as better than what was before), it still feels like an organically grown accumulation of edge cases that were patched. I try to keep it's use to the subset of features that are easy to grow for not just myself but those of my team, and systemd is known (in my circles at least) as being somewhat obtuse. For me, that's a good argument for a dedicated application to handle it.
Can I still get the same networking functionality with undocker/systemd that docker provides? For example, container hostname resolution, private networks, container->localhost port forwarding?
That's really interesting, I had no idea systemd could do that. Have you written in more detail anywhere? I'd love to see a detailed comparison between approaches.
Wow I must be quite behind on systemd indeed. It's been a few years. Is there any isolation offered like you would get with docker or is that a tradeoff?
Edit: looking at the docs is that system-nspawn that's actually doing the heavy lifting. For Linux namespaces. I feel like this has great potential!
I actually use systemd, not systemd-nspawn (it's also in the README), because using both does not always play together.
As far as systemd vs dockerd goes, dockerd provides a bit more isolation by default, but `systemd-analyze security` can guide you so far beyond the Docker defaults. And it is always compatible with system daemons, giving consistent configuration between, say, postgresql from package manager and prometheus from docker container+undocker.
> Goal: no more daemons to run 3rd party containers, systemd is good enough by now: resource limits, isolation, chroot, dynamic users, logging, and more.
Isn't that what Firecracker[1] does ?
(Well, ok, not really, but a bit closer to the base system than Docker/Podman)
Ive been using Podman for a while for my non-Kubernetes cloud deployments (i.e. small VM based things).
It's worked very well for me after a few initial hiccups a year or so ago.
Now that Podman-compose[0] is in the works, it'll really be comparable in the UX space soon, and outperforms Docker in several ways when it comes to security.
The key difference with Podman compared to Docker is that is does not run a deamon as root, like Docker does, thus all containers are created with the privilege level of the user who created it.
This can be a learning curve for those used to Docker as privileges (e.g. for filesystems, files) and capabilities (e.g. for devices, low level networking) need to be handled more explicitly as opposed to Docker's approach of "simon (root) says".
Additionally, Podman is very light weight due to the lack of a daemon since there is no service or supporting software which needs to run beyond the capabilities baked into Linux.
Podman [supports][1] `docker-compose` now. Just set `DOCKER_HOST` to the path of your Podman socket after activating the service, and it should just work (unless you use Swarm, which is not supported).
It's worth noting for others that (it appears from a quick read, I haven't actually used this yet), the compromise for gaining the "docker-compose" superpower is that you will have to run a podman service (Daemon). This comes counter to some (not all) of the benefits I mentioned above, but is a necessary compromise if one wants the power of compose style orchestration; that is, that there must be some deamon to manage it.
This is not authoritative, I may be mistaken, but this is my educated guess based on a quick read and my knowledge of Docker et al.
The Podman daemon can run as a user service, so the only advantage that would be lost is not having to run a daemon at all (but I don’t think one can avoid that if Docker API compatibility is needed). Is there something else I’m not considering?
No Daemon at all is what I was alluding to. A smaller memory footprint and lower attack surface are advantages of that, it may not be an issue for many/most but is worth pointing out I hope.
Having just fought it, Docker rootless is a pain to set up and feels like a hack; it's not the default behavior, requires a lot of additional setup to get it working, behaves differently than rootful docker, and lastly most documentation assumes you're using rootful docker because it's been the only way for years.
The fundamental architecture of docker makes rootless awkward but the company needs to compete with podman now which is architected to fix many of Dockers deficiencies while maintaining it's many strengths.
Docker has been a great tool but running as root has always bothered me. I'm glad they're evolving but it feels a little too late and the migration to rootless, as far as I'm concerned, is not simple. Currently I'm investigating migrating my homelab and development efforts to podman.
The idea that Podman is "more secure" than Docker is hyperbole. As documented in Arch Linux [0]...
"Podman relies on the unprivileged user namespace usage (CONFIG_USER_NS_UNPRIVILEGED) which has some serious security implications..."
A lot of hype around Podman on HN this week (two FP conversations regarding, but nothing "new" with Podman). Seems to be an intentional push to get people talking about it for not much of an apparent reason.
I just found out about Docker's rootless mode in an HN thread the other day. The docs make it seem simple, with few meaningful limitations. Really interested to hear some more about the additional setup you needed to do, and what behavioural differences you've encountered?
They recommend using Ubuntu. And they are not joking. Just click on the other distros to see the amount of hoops you have to jump through. You can't even get overlay2, which offers the best filesystem performance. Not to mention the benchmarks I've seen have slirp4netns at about ~3% of the performance of root veth. I would consider both of those incredibly meaningful limitations. You have to use sysctl/setcap to get ping working, to bind to ports <1024, muck with systemd to get user processes starting at boot, etc. etc.
It's a tough sell when you can just install root Docker with a single package command and never have to worry about a bunch of caveats that might just break or change on the next release.
It's not that big a deal doing it from scratch, but most users of Docker were already using it and have it setup to run as root. Undoing that is not trivial, definitely not as simple as just uninstalling and reinstalling rootless. You have to nuke your existing containers, images, and volumes, and depending on the package manager, it's not always obvious if uninstalling Docker will undo all the networking and user configuration changes it makes. Just out of curiosity, I uninstalled it on Arch right now and it didn't get rid of the docker0 virtual bridge device, it didn't remove the iptables rules, /var/lib/docker is still there with everything in it. The docker socket is still there, but I'm guessing that would go away if I rebooted, assuming the systemd unit creates that on login.
Extra packages/steps/incantations that to be performed/installed, introduction of a docker context mechanism that requires understanding, lingering inconsistencies with external filesystem mounting as root owner, incompatibility with most existing docker scripts and compose scripts, confusion on how to get back to root mode, etc.
It just didn't feel like a turn key solution. I have no idea how this would work in CI/CD systems though docker doesn't always need to be as secure there. Docker is a great tool but, like a lot of tech, in the mad rush to market, security was an afterthought and nowhere has it felt more clearly to me than in rootless mode.
I do not agree that the architecture makes it difficult to do . The main issue is you have to start the daemon. Systemd can at least manage this for you as an unprivileged user.
That said, certainly a no-daemon approach takes an extra step out of the mix here.
Although true, podman did have this feature first, and it's still experimental and unsupported in docker. Docker provides scripts to attempt the setup automatically if you're on a Debian or Red Hat based system, but if you're using something else, you're on your own, and it's complicated and error prone to do, mostly because configuring the daemon and socket to work without root is way more complex than just adding in UID/GID mapping and installing slirp4netns, which is all you need to do to get rootless podman working. It's even easier now that overlayfs works without root without needing fuse-overlayfs, but you're kind of aced out of luck on the docker side with them only supporting Red Hat and Debian, since I don't think any of those get the latest kernels like you'd get with Arch or something you just customize yourself, other than maybe Fedora Stream?
Is that a new podman-compose? I've been using a script from somewhere else, but it drives me nuts since ctrl-c doesn't stop the containers, just the script. Gonna give this one a try; thanks!
> They will get there eventually but this is way over hyped for a sensible replacement for docker on Mac.
The nice thing is that even if it's not useful for you on your development system, your Docker work will still run on it the exact same elsewhere where it does work (hopefully).
There is a somewhat recent article from Red Hat on how to get it working on a Mac[1], but it seems somewhat convoluted in that a lot of stuff (virtualization) that Docker for the Mac apparently does for you is not automated yet.
Edit: Oh, I guess this announcement for 3.3.0 is more/better non-unix support? Is that the version you tried? Maybe they made some of this easier.
Seems like the majority of responses I get when I ask about Mac support is "why don't you just run Linux?"
I do, but not for my workstation, where I do media production and a lot of other things that require a GUI and I use software that, unfortunately, doesn't run as well on Linux (if at all).
I have plenty of Linux servers, but if I wanted to do all my development work 'over the wire' I wouldn't use Docker Desktop either.
I installed Podman 3.3.1 on a Mac yesterday from Homebrew and volume mounts were definitely an issue, but it otherwise worked as advertised on the command-line.
The bugs cited in the article are already fixed in the code, so I don't hold those against it.
I did have an issue that was my fault, encountered one genuine bug, and was disappointed to find that there's some work to be done before third-party tools can use it as a Docker replacement:
I agree that Podman is not yet a complete replacement for Docker on macOS, but the experiment was successful enough for me that I intend to try Podman instead of Docker on Linux servers.
On CentOS and similar distros I agree. But on Debian like distros podman has too many rough edges and the packaging isn’t something you can contribute to easily on GitHub. I find this sad since podman is really good but there isn’t much effort in improving UX on Debian distros.
How do you handle load balancing of inbound traffic.. do you use a pod running Traefik or similar? How do the pods communicate when they are deploying, unavailable or busy and so on?
I guess I could get this from your site but there's a LOT of information on the first pages there and possibly not what I am looking for.
We use haproxy pods for inbound traffic, they perform TLS termination and simply pass the traffic over to a local proxy pod.
The haproxy pod (and all other pods) communicate with each other over this local proxy service which is running on each host.
We have a very simplified and robust overall architecture, were each pod allocates a specific virtual port and the proxy will try each host for that pod (and remember status), meaning we don't need to keep track of global routing tables and update each host's ip tables (shivering) when pods come and go. We don't use iptables.
If some instance of a pod is unavailable the proxy will seamlessly try another instance of the pod.
Initially we tried to config haproxy to do all this proxying for us, but it was asking too much.
Good to know is that Simplenetes is still in beta.
Indeed, the goal has to keep the number of moving parts down as much as possible so it can be easy to understand the full cluster and how to troubleshoot it. But of course, it still requires knowledge about the architecture to do so.
How was your experience writing that much Bash code?
I wonder what tools there are, currently (I noticed it's in beta), to get an overview of the state of the cluster, maybe what is talking to what, how much bandwidth they use etc (I don't know what one would need to know)
Thanks for asking :)
Writing this much Bash is quite straining because there is a lot of typing, but it is also liberating in the sense of coding very close to the OS (utils).
Also, most of it is not written in Bash, it's written in Posix standard, which is even more spartanic, but is then compatible with Dash and Ash (BusyBox) also, which is good because Bash is not always available.
To make Simplenetes we used another tool we also created which is meant for writing shell script apps and to perform agent-less automation, it is called Space.sh [1]
About tools for getting an overview of the cluster, there is only the command line tooling as for now, which does parts of the job, but tools for analyzing traffic and such is not created yet.
Lima[0] also looks like a promising replacement on macOS. It manages provisioning a VM with nerdctl[1] installed, networking, and shared storage. The project has a support path for ARM Macs with a patch QEMU.
Someone correct me if I'm wrong but my understanding is that Docker is only charging for Docker Desktop, which is a GUI and management tool required to use Docker on Windows and MacOS - both of which are not supported by Podman. And Docker Desktop is not required (or even available) on Linux - the only platform Podman supports.
you're right. And the caveat is still that Docker Desktop costs $240 a year only for companies with $10m+ in revenue or 250+ employees. Free for everyone else.
I don't get the whole "Let's spend weeks rewriting our build infrastructure to save $240 a year" strategy going on here.
$240 a year * 250+ employees is 60k at the very least.
My company has over 3500 engineers. That's almost a million dollars in new spend and you gain basically nothing. It's a product the entire org has already been using for a long time, now you just have to pay a million dollars a year to use it.
You gain support. And the continued existence of the company that developed Docker CLI & GUI. Surely that’s important to your 3500 engineer company’s future too?
> I don't get the whole "Let's spend weeks rewriting our build infrastructure to save $240 a year" strategy going on here.
I think that's because 2 things are at play here:
1. People are busy and often rarely read past the headline announcing the change, so they are unlikely to know of the caveat.
2. People don't like being taxed at the infrastructure level, so the default price for plumbing tools like Docker is $0. What's the assurance that the $240/yr will not rise to $2400/yr a year later, especially if they fail to hit their revenue numbers? Once the glove comes off, it's so much safer to cap your costs today by seeking out an alternative, rather than be confronted with uncapped costs (or any kind of vendor uncertainty) in the future.
I think there’s a third problem. Docker on MacOS is still pretty buggy. After all these years there are still issues that cause it to use 30% CPU when idle on my Mac, and I’m sure I’m not alone. If docker on Mac was rock solid then I’d probably look more kindly on this.
We aren’t even big enough for this to affect us, it is just one more reason for us to look at how we can remove docker from our desktop infrastructure and replace it with something that makes us more efficient.
> After all these years there are still issues that cause it to use 30% CPU when idle on my Mac, and I’m sure I’m not alone.
I had the same frustrations until the other day I saw a tip to enable 'Use the new Virtualization framework' under Preferences -> Experimental Features. Since then Docker's background CPU usage has dropped to 1% and I no longer bother stopping it when I'm not using it.
I definitely share your thinking on the costs but there is one complication: purchasing things can be surprisingly hard in large organizations, and if Docker doesn’t have the blessing as an enterprise procurement that can be far more in overhead than the base cost.
For people in .gov, that can be waiting a fiscal year to get it in the budget and slogging through paperwork to show that they’re in compliance:
Also, build infra doesn’t even have to change. Only development tools. I just uninstalled Docker Desktop and installed minikube (single binary) and I’m all set.
I don’t really get it — my low resolution understanding is that Nix gives you the ability to define your Unix environment itself with definition files, similar to how Docker gives you the ability to define a container abstraction with Dockerfiles.
So if your whole machine’s environment is already specified, why do you need to add Docker as another layer of abstraction? Is it simply to deal with needing to run multiple apps on the same underlying machine (defined by Nix) with possibly conflicting dependencies (those would be managed by Docker)?
With nix, the whole dependency hell problem is solved, so you will not get conflicting dependencies anymore (other than perhaps bugs). You could install the same program twice with the exact same version of libc compiled by gcc and clang for example and it would still not get into each other’s way.
Docker on the other hand doesn’t solve this problem itself - docker images can be built declaratively (eg. with nix itself), but it is more about managing the running of services with possibly different environments. A typical dockerfile will not be identical at all between two separate creations (most of the time it just installs programs from a repository without any versioning other than perhaps the distro’s major version)
The fact that we as in software developers use it for dev environment is just the unfortunate way it is.
You can get the reproducibility resolved. But when I tried nix I didn’t found a way to save the source code easily in my project. Similar to the vendor directory in Go there should be a way to backup the source since the source can go away any time.
What do you mean? Just use a git repository as a source, but you may as well create a hashed folder containing your source files, but that is not the primary function of nix.
I mean so I can reproduce it in 10+ years where many git repos don’t exist anymore. Go can keep checksumed copies of any code it uses to compile. I’d like to see this in Nix as well.
One use case I see is if you're using Nix for all your local development, but Docker in production (because it's the easiest these days: package an image, give it to ECS/Fargate or equivalent and it's good to go).
You're not using Docker in development because of the sync issues on Mac between the host and the containers, but ideally you still want as much as possible the development and production environment parity (same versions of dependencies, etc). You can build your Docker images with Nix to ensure the dependencies versions you're developing with will be the same in prod.
I noticed in the documentation it says non Linux as an option. Is anyone using podman to run Windows containers? If so, have you written it up? I've exported a bunch of containers from Google Cloud to run on virtual box but it's clunky and doesn't always work.
How does podman replace docker desktop? I thought it more as an alternative to the docker daemon, which is still free and open source. I start to wonder if people actually value open source if they run away from something just because some company publishes a commercial GUI.
I like that with docker swarm one can without too much headache go from dev environments to a cluster. At university this seems much better than maintaining k8s. Podman seemed to be more interesting in rootless environments like HPC to me. I really what benefit switching to podman would actually bring to us.
Using any sort of Linux container runtime on Mac requires a VM, since you can't otherwise get the Linux kernel features required. Docker Desktop and podman machine will both do the VM provisioning for you, and then you can use docker the container engine or podman to actually drive containers in the VM it gives you. So it effectively can replace it because podman machine will provision a VM for you, but you can of course use any other VM provisioner you want. There is no need to have it coupled to the container engine.
It's worth noting if you have an M1 Mac, podman machine doesn't work on it yet. Docker Desktop does.
Of course, the easiest way to use containerization technology if that's what you want to do is just use Linux workstations and don't worry about needing a kernel virtualization layer between your workstation and your container engine at all.
Edit: Noting since sibling comments are discussing Windows and WSL that I'm talking specifically about replacing Docker Desktop on Mac because the article linked here is talking about using podman machine to replace Docker Desktop on Mac.
It is straightforward to get Podman up and running in WSL2; it is also possible to set up Docker in WSL2 as well without Docker Desktop as well but it is much more painful. So in the Windows space, Podman effectively becomes an alternative to Docker Desktop, simply because the latter makes it possible to do Docker at all (ignoring the other things it does).
I just noted that it seems to straight forward to extend machine via drivers and at least there is one for parallels. Also rancher seems to maintain a fork of machine itself so it seems unlikely to die as an alternative...
podman has a `machine` subcommand for setting up a VM for running podman. It configures the podman server and your podman client connects to the server, very much like docker.
I’m not sure if the API is stable enough. But podman is in fact libpod (was named to podman) and it’s used to some extent in Cri-O. Nothing prevents you from using it as a library.
podman is wholesale copying docker, doesn't that feel a little bit unfair? What does docker think about it, I guess they don't want to draw attention to it(?).
Offering API-compatible tools is very deeply ingrained in the UNIX philosophy; just think of the different implementations of "ls", or the C functions mandated by POSIX, or the way that clang (the compiler targeting LLVM) has the same flag names as GCC, which probably copied them from somewhere else in the first place...
Generally, if you publish an Open Source project, you cannot complain if people copy parts of it, even it's the command line arguments / API / whatever.
One can also argue that the “openness” is what allowed docker to be popular. If it was locked down from day 1 there’d have been a lot less interest. Nobody likes being tied to things at that level.
Goal: no more daemons to run 3rd party containers, systemd is good enough by now: resource limits, isolation, chroot, dynamic users, logging, and more.
Low-level tooling is done, I am now building ecosystem around it: easy installation, convert to deb/rpm, systemd units, etc.
[1]: https://sr.ht/~motiejus/undocker