We already help with automating jumpbox creation. (docs: https://userify.com/docs/tips/jumpbox/) and we're building even more jumpbox automation now. Don't allow root for any jumpbox accounts, but of course root escalation exploits abound. (I just found another in an AWS agent yesterday.)
Of course, as another commenter mentioned, if your jumpbox is compromised, than the jumpbox could serve as a gateway to your network. It's a tradeoff between exposing all of your servers to inbound SSH or only one.
There's another way, which is pure TCP forwarding on a different port for each server (ie 21321 -> inner server 22), but whether this actually reduces the attack surface is debatable, since the totality of open ports remains the same across the entire network.
My personal feeling is that using a jumpbox and locking it down (preferably to your company's IP ranges, etc) is the best way to go. You can also add MFA to the jumpbox entry point itself. We're going to help with automating all of that in the near future, too.
One thing is setup of jump hosts (basic version just being a box with sshd enabled + users with authorized_keys, which is beyond simple), but it's the limitation of hosts. I have 1 big network where a set of users are only allowed to jump to 1 or more unique host each, with myself and a few others being able to jump to every single one. I believe that your solution requires things to be on different subnets to isolate them, with separate jump hosts for each subnet, correct? Here, I only need one sshmux for everyone.
Another "fun" feature is that sshmux can also throw unknown users to a separate host, such as ssh-chat, for support or hanging out.
Actually, in case of -oProxyCommand="ssh -W %h:%p jump_host", you're still fully secure in the case that the jump host (sorry for not calling it a jumpbox) is compromised. This is true both for normal SSH servers and for sshmux. They can mess with the original raw ssh connection that requests the forward, but this will just screw the connection up. They can connect it to a wrong host, but that will provide incorrect host keys. The agent isn't forwarded, so they don't get to sign anything with the private key. Messing with the new connection that gets established is subject to SSH's normal MITM resistance. This means that even root on the jump host would require real SSH protocol vulnerabilities to attack the connection.
That's cool. Another way is to localtunnel to TCP forward to endhost in your config (ie two SSH connections, the first to the jump host, the second through the jump host, without agent forwarding), but that gets tedious fast without scripting support, especially if you want to use different portnums to avoid conflicts. ControlMaster would be a good choice to multiplex multiple connections to the same box.
> I believe that your solution requires things to be on different subnets to isolate them, with separate jump hosts for each subnet, correct? Here, I only need one sshmux for everyone.
Right, we're automating the .ssh/authorized_keys (and sudoers) right now (https://github.com/userify/shim/blob/master/shim.py), so you can set it up however you like, as long as those boxes can get to either Userify for key updates, or your on-premises Userify server(s). Only one jumpbox is needed across any network, depending on how you prefer to lockdown your network. (The HOWTO refers to EC2, but it can be built anywhere, exactly the same way.)
sshmux looks like an interesting project, and love the simplicity. Wonder if Userify might automate the ProxyCommand setup! Seems really useful.
> Good luck with Userify!
Thanks! We have 350 companies on the platform right now and growing fast, and Userify Enterprise (on-premise) is in beta at several large enterprises as well.
BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me if you get bored :)
> sshmux looks like an interesting project, and love the simplicity. Wonder if Userify might automate the ProxyCommand setup! Seems really useful.
Yeah, ProxyCommand is a bit cumbersome for many setups, which is why the agent forwarding mode has the interactive selection screen...
If you feel like poking at sshmux or think there's anything that would be helpful if added, don't hesitate to add it to the issue tracker. Real-life use-cases get higher priority.
> Thanks! We have 350 companies on the platform right now and growing fast, and Userify Enterprise (on-premise) is in beta at several large enterprises as well.
That's great! I guess getting Userify Enterprise out of beta is a bit thing on the roadmap.
> BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me if you get bored :)
It sounds very interesting (I like Go and SSH, what can possibly go wrong?!), and I am indeed occasionally bored, but I'm ~8000km away... Commuting from Sweden to Denmark is bad enough! ;)
> sshmux, and by extension, sshmuxd, can only forward normal sessions (ssh'ing directly to sshmuxd without a ProxyCommand) if agent forwarding is enabled.
This is very dangerous.
With the average setup, anyone with root access on the middle box can borrow the ssh key of any user connecting through it.
Agent forwarding can be dangerous, yes, in the sense that you are giving the remote host you are connecting to (or jumphost in this case) permission to sign things with your private key.
In the normal case, this is done through a socket, which means that root on the machine can access it, and sign arbitrary things. A compromised machine is therefore dangerous. This is no different than getting root access on the local machine, as you can access your local ssh agent there as well.
However, with sshmux, the agent channel is not forwarded as a socket, but instead internally handled as the ssh channel it really is. This means that compromising the agent at sshmux requires arbitrary memory access, which is a considerably more difficult feat, and means that the machine is 100% compromised regardless.
Statically linked, but you can replace the entire application. You'd get one or two private key signa before the user disconnects and starts wondering what happened (remember that you need to have something to sign).
Of course, without askpass, a properly patched evil sshmuxd could sign a separate session while still signing ths original.
As I have said many times, ssh -W ProxyCommand makes you immune to evil jump hosts, but if root on a machine in your network got compromised, as would be required, then you're going to have a bad day regardless.
You need arbitrary read and write memory access to abuse it (how would you otherwise make the signing request? You can't just steal one, as they're unique for their purpose), not just read, which is much more than heartbleed was capable of. You of course have this with root, but I'd like to point it out regardless.
I wrote it because, in the case that you have an evil user that scans all physical memory, you can't do anything on the computer and still be safe. ssh -W bypasses the host, and due to various cryptographic features of that setup, the only thing that could happen would be to have someone break the connection or present you with incorrect host keys, which the client would detect. I'm of course assuming that they don't have a SSH vulnerability, as that's a completely different story regardless of agent forwarding.
You can't use that machine for anything else and still be safe, and the presence of that machine on your network is a bad sign, most likely indicating that other things will be taken over as well in the same way that they got first one, or already is compromised, including some of your endpoints. While agent forwarding to the final host isn't necessary, your network is compromised, meaning that other important thing most likely got stolen, just not your private key. If your trumph card is that they don't have your private key, it's kind of like standing at a house that has been completely destroyed with all valuables in it, holding a key and saying "Well, at least they didn't get the key for the front door!". In this case, having private keys for different purposes would be a good way to isolate things, just in case. I personally separate work keys and private (as in spare time) keys.
I know I'm exaggerating the comparison a bit, but the setup required to take over agent forwarding on a network you control, means that you're screwed regardless. Agent forwarding should be used with care, ssh-add -c being absolutely necessary, but agent forwarding has its uses, and many of the cases where I'd be attacked means that I have more pressing matters to deal with.
Exactly. And yes, Kerberos got it somewhat right, but as someone who have used configured and used it, it is quite the exercise to set up. It took ages to get working, and even then, it was slightly cumbersome to use.
ssh-agent, as a protocol-concept is not as broken as it sounds. It's just that the implemented interface (socket) is bad when combined with no user acknowledgment of sign requests. I think the worst part is that the agent doesn't ask the user for confirmation, which would essentially invalidate most of the ways you would abuse the agent. Attacks would be reduced to spamming or attempted phishing with agent requests.
Kerberos could have bad interface as well, but I don't know how that would work in this scenario. The protocol itself is secure, but just as with ssh agent forwarding, if you let everyone make authentication requests without requiring user interaction to verify it, it will have the same problems.
Personally I have a confirmation dialogue for any agent forwarding request. When I require it, this seems like the best compromise. (I still have my key decrypted in memory, I just get a "Use key?" window that I need to hit enter for).
ssh-add has a -c option, which uses ssh-askpass for confirmation. Unfortunately it doesn't give much information, but you hope you would notice it coming up when you didn't initiate it. You may also be able to do something with gpg-agent, it seems to have more options.
I don't really understand what this is supposed to be useful for.
Relying on a random software to secure your ssh entry point, instead of a proper linux configuration, that seems like a risky tradeoff.
The project is meant to ensure that you can allow multiple users access through a jump-host style mechanism, while not permitting any other "abuse" of the jump host. You can lock down SSH a lot, but not as much as sshmux does. Security wise, the code is very, very simple and easy to follow, and even if it went rogue, that's no different than the trust you put in your usual SSH server. If this is a concern, do not use the agent forwarding mode, which would render a rogue server a pointless and unfruitful prank.
If you use the ProxyCommand mode, the only thing an "evil" sshmux would be able to do would be to break the connection. It won't be able to fake the endpoint if it is already in your known_hosts, as it does not have the real endpoints private host key. It is, therefore, secure in this mode.
When I use SSH as VPN, it's usually because I want less flexible applications to use it, using socks5 configured as a system-level proxy. When doing more work on our corporate network, I usually SSH through the jump to my desktop and work from there, unless I specifically need to access another machine for something.
As for sshuttle, it's a very convenient device, although the design appears rather complicated. I'll look into it.
The design is possibly the simplest for what it does (and it does that exceedingly well).
sshuttle is:
- a tcp multiplexer (multiple connections onto one stream)
- a router (uses firewall rules to make normal connections go through the multiplexer)
- a few kludgy tunneling operations, to make the combination above seem like a real VPN (discovery of remote subnets, tunneling of DNS requests, ....)
- a default setup that runs the multiplexed stream through an ssh connection, thereby giving all the security guarantees that ssh provides (integrity, confidentiality, mitm resistance)
- packaged in such a way that the remote side needs to have a minimal python>2.6 install, and a user capable of making tcp connections. Nothing more.
In my experience, it works way better than IPSEC tunnels and some commercial VPNs that I've used. With two caveats that I'm ok with with: a) only TCP, and b) all connections seem to come from your remote ssh server. No VPN solution that I'm aware of makes so few demands; do you know of one that doesn't require root at the remote server?
ssh -D (socks5 mode) doesn't require root, and can forward both udp and tcp traffic, doing what it appears that the client requires. socks5 is just a protocol that tells the server to connect to X on port Y using protocol Z, or to bind to X on port Y using protocol Z. Root is only required if you want to bind to privileged ports. A bonus with this is that it only requires a SSH server in default configuration.
SOCKS5 most common usage is to just use it as a proxy for HTTP traffic or similar, but it can make any connection. The trick lies in the "kludgy tunnelling operations" and the "router" (firewall rules).It's this magic that makes it better than the socks5 proxy solution, as it really does appear like a VPN. I wonder how much effort it would be to get rid of some of the "magic", though.
But I have to agree, that ANYTHING is better than a real VPN in complexity.
> ssh -D (socks5 mode) doesn't require root, and can forward both udp and tcp traffic
No, but it requires a SOCKS5 client, of which there aren't many among software I use.
> The trick lies in the "kludgy tunnelling operations" and the "router" (firewall rules)
The other important part of the magic is the packaging. While ssh+socks support is pretty common these days, it is not supported on e.g. dropbear (which is popular on routers).
sshuttle copies whatever it needs to the other side; as a result, there's never an issue of version compatibility (which socks use sidesteps by virtue of using an old, cemented protocol). I have, in fact, needed to modify sshuttle with new options and features for a project, and I was not bound in any way by the existing protocol - whatever changes I made were always supported by the other end of the connection because the connection starts by copying them to the remote.
> I wonder how much effort it would be to get rid of some of the "magic", though.
sshuttle is very well written; It is easy to tear it apart and rebuild it in other ways. But if you took the magic apart, you'd be left with a non-standard UDP-deficient SOCKS-alike system. Why would you want that?
If you tied the local FW magic to a socks5 client, you'd be able to skip part of the magic, assuming you don't desperately need mDNS. SOCKS5 does UDP too, IIRC. I like tearing things apart, and I like making clean muggler solutions, rather than relying on wizardry. :)
With that said, I might still end up using sshuttle as it is. It's a nice project. I just crave for tinkering around. :D
sshmux is a jump host that allows more user-control than you can with a ssh server. Unless you use the interactive mode with agent forward, you still need to configure your client to use one.
I'm not sure I get what you're on about, to be honest. :/
SFTP/SCP works if you use the ProxyCommand, or agent forwarding. Clients such as WinSCP support agent forwarding, at least. Not sure about FileZilla. I'm not a Windows user, so can't say. :/
I know FileZilla also supports agent forwarding, but I'm not entirely sure how this would work in practise. If I'm understanding things right you can have a single jumphost and give a user access to multiple targets. So when the user connects to the jumphost using SFTP then what files/directories do they see?
Maybe I should just install and try this for myself...
I see why you might think that, but no. SSHProxy tries to move the private key from your computer to an external thing. SSHProxy doesn't have anything to do with sshmux.
SSHProxy is a two part thing:
1. A server that, when connected to, will ssh to the requested host, but authenticate with a local private key.
2. A client that will, when connected to the server, work like netcat for ProxyCommand purposes.
I'm skipping some info about additional SSH sessions being involved, as they don't provide any additional security.
The result is essentially ProxyCommand="ssh -W %h:%p jump_host", but instead of you having the private key, the jump_host has it. You then use a different private key to authenticate to the jump host.
I get the idea, and I by no means say he shouldn't have implemented it, but this provides no additional security what so ever. It just moves your private key from one machine to another. I hope he had fun implementing it, though! If you don't implement what might be a bad idea, you'll never get around to the good ones. Plus, bad ideas can be fun, even if they remain bad when done.
Ah yes, on closer inspection on a non-phone they are quite different.
You are just wrong on the no additional security though. In addition to enabling auditing, it also allows a username/password to be turned into an ssh key, so that you can know that if the rpi wasn't turned on, then nobody could log in to the target machine. It's similar to a TPM in this regard (as blog says), or can be a jumpgate protecting the systems behind it from compromised or malicious users.
I'm sorry you don't get it, but I'm not hurt by someone not understanding something when they criticize it. If you don't see the point of not having direct access to your key, then there's nothing I can do.
1. The private key is not secure by default by putting it on a different host.
2. This still allows an evil user getting access to his original machine to SSH into any host requiring his private key.
1 should be pretty obvious. If my intention is to get a hold of the actual private key file, he just moved the target.
As for 2: We're talking about avoiding compromising a private key file stored on the client, so in order for that to be an issue in the first place, we're giving an evil user Y reading abilities on the local client. The normal user is called user X. The evil thing to do is to log into the other server on poor user X's behalf. User X used SSHProxy, and moved his private key to a raspberry pi. So, what does evil user Y do on user X's computer?
He just logs in through the SSHProxy, getting access to the remote service. Having the raspberry pi disconnect at times doesn't really matter, as that just means patience (or starting a job that keeps trying). Once logged in, evil user Y could add another key to the authorized_keys file if he wanted to.
Having a service automatically sign things with private key A stored in remote location B, as long as you provide private key B stored in "easily accessible" location A (Which we still assume due to thinking the private key was insecure) is no more secure than storing private key A in "easily accessible" location A. The only argument becomes the ability to occasionally take the pi offline, which doesn't seem to be what he touts as a reason for the project. The issue with this is essentially a reversed modified version of the ssh-agent socket issue, which is blindly signing things as well.
If you truly want to protect your private key:
* Encrypt a USB dongle
* Put a password protected private key on it
* Only plug said USB dongle in when you need it, store separately from your laptop (on your person)
* Change your password occasionally
Likewise, having a TPM be your signer doesn't change anything if you have it sign blindly. PCR's allow you to at least verify system state first, but using the TPM in this case only protects you against people stealing your filesystem without your physical machine. With access to the machine, the TPM will keep on signing things blindly. User interaction beats TPM in all areas. One of the cute features TPM was supposed to bring, that actually was going to matter (if you ask the manufacturers, that is) was Remote Attestation. Unfortunately, it wasn't any good and was never used for anything.
Also, for future reference, while it's completely okay to suggest that people are wrong or misunderstanding, there are nicer ways to say it than "You are just wrong", and "not understanding something when they criticise it". I have no idea about your qualifications, and you have no idea about mine. We're all humans, so be polite about it. No one likes being insulted.
> Also, for future reference, while it's completely okay to suggest that people are wrong or misunderstanding, there are nicer ways to say it than "You are just wrong", and "not understanding something when they criticise it".
It's useful information to me that you value pleasant phrasing over criticism that might improve your security.
It's also good to know that you completely ignored security advice from security professionals, making this a waste of time. You could say that up front as well: "I'm going to write a comment insulting you, but don't you DARE try to disprove me with logic!"
I've used a similar setup to wrap RDP to allow employees to securely remotely access their office desktop. There is a market for someone willing to make this a turn-key replacement for Terminal Services.
Probably 15 years ago I did something similar at an overall system level with a program called pdmenu which provided a jump host menu for semi-technical users. The technical details of course are entirely different, but it did the same general system idea of presenting individual users with custom tailored prompts to log into various systems (and a few other tasks, and logging, etc). pdmenu had (has?) a great menu CLI for end users.
Landskrona, actually. Malmö would have been nicer, but the waiting list for an apartment is 2-3 years.
It's actually quite common, especially in Malmö (Malmö<->Copenhagen takes about half an hour, Landskrona<->Copenhagen 1:20), for a lot of reasons. The first being that living expenses, including rent and household items, are cheaper in Sweden. Local wages are always adjusted after the local living expenses, so living in Sweden while working in Denmark is economically favorable, especially seeing that you get a tax deduction bonus based on distance from home to work through the shortest feasible route. Cars are also orders of magnitude cheaper in Sweden due to different taxing. Not everything is cheaper or better there, though. Diesel and alcohol is more expensive, and my selection of rye bread is much narrower up there!
For me, it's due to me wanting to start a life with my wife, and a lot private things going on that made Sweden the obvious choice.
This is nice but the agent forwarding is a legitimate concern especially because a compromised host can take off with the private keys which are used while beginning the session. Most private keys are used for more than connecting to a single host.
In our setup we use jailkit allowing only ssh passthrough [1], we have added LDAP support to it (may release the patch later).
What authentication do you have from the jump host to the target? To me, it looks like it has been reduced to either none or keyboard-interactive (password) login, which is considered bad practice. I could very easily implement support for this in sshmux, providing the setup you seem to use, as a way to avoid agent forwarding, but I just genuinely did not expect anyone to use password authentication, apart from in default config scenarios, before a public key has been installed.
Agent forwarding does not provide private keys, but only individual signing requests. This means that while the user is connected, and evil remote can request arbitrary signing, but only as long as the user is connected, and only as long as the users ssh agent is willing to do so. Using ssh-add -c further means that the user will have to accept each signing request. Also note that ssh -W, which is immune to any of these concerns, is fully supported by sshmux.
Correct me if I'm wrong, but "Jailkit" does not seem to stop an evil legitimate user from poking around other hosts with forwarding and such (which they can as long as ssh -W functions, within the limits of what a firewall allows the jump host to do in general). sshmux allows you to lock users to targets, rather than lock general capabilities of the entire jump host. This is because, while I provide clients access to various hosts, I do not trust them enough to have any unnecessary privileges.
-W is essentially the same as using nc, however it doesn't require executing a command on the jumphost. This project uses that fact to make the jumphost more secure (you can't execute arbitrary commands), and do destination white-listing for each user.
sshmux gives fine-grained user controls. In the normal jumphost example, you can use ssh -W to connect to arbitrary ports and hosts on the network, poking around where you weren't intended (ssh -W is just netcat where the ssh server initiates the connection for you). With sshmux, ssh -W will only work to hosts permitted for that user, and any other request will just fail.
That's true. Some people complained that this isn't documented enough, so there is a wiki page for sshmuxd explaining what it does.
Agent forwarding with sshmux should be safer than the general case, though, as the agent is handled in memory, rather than as a socket. The agent is also not forwarded to the destination.
VPN's can be quite a pain, and is considerably more work than raw SSH. Depending on the VPN, the authentication strength is usually also quite a bit lower than that of SSH with RSA keys. There's also additional overhead. I have both VPN and jump hosts to get into our corporate network, and I most certainly prefer the jump host for convenience and performance.
There is also the user-level access restrictions of sshmux which will be much more difficult to replicate with VPN's. At least with pptpd, I believe it would require putting users on individual subnets with firewall rules restricting access to only their permitted hosts. Long story short, it wouldn't be a feasible solution.
We already help with automating jumpbox creation. (docs: https://userify.com/docs/tips/jumpbox/) and we're building even more jumpbox automation now. Don't allow root for any jumpbox accounts, but of course root escalation exploits abound. (I just found another in an AWS agent yesterday.)
Of course, as another commenter mentioned, if your jumpbox is compromised, than the jumpbox could serve as a gateway to your network. It's a tradeoff between exposing all of your servers to inbound SSH or only one.
There's another way, which is pure TCP forwarding on a different port for each server (ie 21321 -> inner server 22), but whether this actually reduces the attack surface is debatable, since the totality of open ports remains the same across the entire network.
My personal feeling is that using a jumpbox and locking it down (preferably to your company's IP ranges, etc) is the best way to go. You can also add MFA to the jumpbox entry point itself. We're going to help with automating all of that in the near future, too.
(disclaimer: CTO @Userify)