> was always told early on that although they supported vault on kubernetes via a helm chart, they did not recommend using it on anything but EC2 instances (because of "security" which never really made sense their reasoning).
The reasoning is basically that there are some security and isolation guarantees you don't get in Kubernetes that you do get on bare metal or (to a somewhat lesser extent) in VMs.
In particular for Kubernetes, Vault wants to run as a non-root user and set the IPC_LOCK capability when it starts to prevent its memory from being swapped to disk. While in Docker you can directly enable this by adding capabilities when you launch the container, Kubernetes has an issue because of the way it handles non-root container users specified in a pod manifest, detailed in a (long-dormant) KEP: https://github.com/kubernetes/enhancements/blob/master/keps/... (tl;dr: Kubernetes runs the container process as root, with the specified capabilities added, but then switches it to the non-root UID, which causes the explicitly-added capabilities to be dropped).
You can work around this by rebuilding the container and setting the capability directly on the binary, but the upstream build of the binary and the one in the container image don't come with that set (because the user should set it at runtime if running the container image directly, and the systemd unit sets it via systemd if running as a systemd service, so there's no need to do that except for working around Kubernetes' ambient-capability issue).
> It always surprised me how these conversations went. "Well we don't really recommend kubernetes so we won't support (feature)."
> Me: "Well the majority of your customers will want to use it this way, so....."
Ha, I had a similar conversation internally in the early days of Boundary. Something like "Hey, if I run Boundary in Kubernetes, X won't work because Y." And the initial response was "Why would you want to run Boundary in Kubernetes?" The Boundary team came around pretty quick though, and Kubernetes ended up being one of the flagship use cases for it.
Thanks for the detailed explanation - some of what you say sounds familiar, but this was nearly 5 years ago so my fuzzy recollection of their reasoning - I recall it being something like they didn't trust etcd being compromised on kubernetes. My counterargument to that internally was "if your etcd cluster is compromised by a threat actor you have way bigger problems to worry about than secrets"
My vague recollection is that that concern was that the etcd store (specifically the keys pertaining to the Vault pod spec) could be modified in some way that would compromise the security of the encrypted Vault store when a Vault pod was restarted. It's been a long time since I remember that being a live concern though, so I've mostly recycled those neurons...
The reasoning is basically that there are some security and isolation guarantees you don't get in Kubernetes that you do get on bare metal or (to a somewhat lesser extent) in VMs.
In particular for Kubernetes, Vault wants to run as a non-root user and set the IPC_LOCK capability when it starts to prevent its memory from being swapped to disk. While in Docker you can directly enable this by adding capabilities when you launch the container, Kubernetes has an issue because of the way it handles non-root container users specified in a pod manifest, detailed in a (long-dormant) KEP: https://github.com/kubernetes/enhancements/blob/master/keps/... (tl;dr: Kubernetes runs the container process as root, with the specified capabilities added, but then switches it to the non-root UID, which causes the explicitly-added capabilities to be dropped).
You can work around this by rebuilding the container and setting the capability directly on the binary, but the upstream build of the binary and the one in the container image don't come with that set (because the user should set it at runtime if running the container image directly, and the systemd unit sets it via systemd if running as a systemd service, so there's no need to do that except for working around Kubernetes' ambient-capability issue).
> It always surprised me how these conversations went. "Well we don't really recommend kubernetes so we won't support (feature)."
> Me: "Well the majority of your customers will want to use it this way, so....."
Ha, I had a similar conversation internally in the early days of Boundary. Something like "Hey, if I run Boundary in Kubernetes, X won't work because Y." And the initial response was "Why would you want to run Boundary in Kubernetes?" The Boundary team came around pretty quick though, and Kubernetes ended up being one of the flagship use cases for it.