I really hate the accusation that changing services to uncommon ports is security through obscurity. It's not a security practice, it's a high pass filter.

There's billions of hosts on the Internet and many millions of them running an SSH service. Even attackers with huge botnets at their disposal have finite resources and time to probe for hosts and then try exploiting vulnerable ones. Every exploit attempt has some percentage chance of being against a honeypot used to help catch the attacker.

There's little point in even sniffing non-standard/uncommon ports for different services. Anyone changing a service port has a higher likelihood of having a more secure system than someone with standard ports. So even if you scan high number ports for an SSH service you're unlikely to find one on a high port that's not using a more secure than default configuration.

So when you have SSH listening on some uncommon port and see hits in your logs you have a reasonable assurance the attacker isn't some script kiddy spamming the "attack" button on some stupid app. You also have reasonable assurance blocking that host/network with fail2ban or equivalent will actually be effective. You can also set way more strict fail2ban rules since you're far less likely to ever lock out authorized traffic. The more specific your allow criteria the better and more effective your block rules can be.

A "Beware of Dog" sign doesn't physically secure your house on its own. You might not even have a dog. What it does in change the calculus for a burglar. Why risk it when there's more obviously less risky targets nearby? The sign is a high pass filter.

So here's the problem:

Ports under 1024 are privileged in Linux - they require root or root delegated privileges to run. When SSH is on port 22, you have the assurance that unless the server is root compromised, it is what you think it is running on that port.

When you make that port 2222 or whatever, like so many people do, you have cut out a lot of noise... but now that compromised PHP application you had running has allowed someone to now race you every time SSH is restarted for an update or crashed or whatever to bind on that port. Let's say it wins - now you've got something listening on the SSH port. If someone using SSH ignores the fact that the host key has changed, now they're trying to login to a fake SSH server. Maybe you use password auth and you just gave them the password. Maybe they're using an OpenSSH client that is vulnerable to leaking private keys in certain situations. Maybe the fake SSH server pretends to be a real shell and they then log whatever actions you try to take when you SSH in. Because they're going to be able to figure out what port SSH is listening on - a fingerprinting port scan can be done in seconds.

You are sacrificing security when it comes to a more focused attacker for the sake of filtering out the low effort mass scans and basic brute forces. The thing is - I'm worried about the former, not the latter.

If you care enough about securing or filtering your SSH to go through any of this trouble, just set up a VPN on a separate machine and restrict SSH access via firewall to that machine. Spin up the smallest VM your cloud provider has, throw up wireguard on it, and you're good to go. It'll be plenty for a VPN that's basically just there for SSH access. Now someone has to have both an exploit for wireguard and an exploit for SSH to get into your machine that has things you care about, you've filtered out all the noise, and you haven't introduced new security risks for a more determined attacker.

What you mention makes sense if all of the wild potential problems exist on the system. You've already fucked up leaving password authentication enabled on your theoretical machine.

It's pretty trivial to leave SSH on port 22 and just forward a high number port to it while blocking 22 externally. All the root-not-compromised assurances and still maintaining the high pass filter.

While a VPN certainly can be a good solution to securing a machine, you've now got the problem of the VPN server needing its access protected.

Public key only for authentication and strict fail2ban rules combined with port forwarding makes for a very tight system. Not invulnerable but secure enough to not be worth the effort.

As I said originally and still maintain, using non-standard ports is a high pass filter. It's not a security measure. It might be part of your security setup but it's just a filter.

It doesn't necessarily need to be a high port either.

All bots will check port 22 first and then move onto others. Your can have a port sniffer watching this and block any attempts to your actual <1024 port of they've also hit port 22.

It works a treat. Shodan, begone. I agree with your high pass sentiment

> It's not a security measure. It might be part of your security setup but it's just a filter.

I think most people saying security by obscurity 1) know that’s a fallacy 2) know that by using that phrase the reader/audience knows what they mean without having to explain a phrase like high pass filter. It’s a bit pedantic to rant on the use of terms.

> You've already fucked up leaving password authentication enabled on your theoretical machine.

You could also be exposed to this with key based auth - older openssh clients have a security flaw where they can leak keys, and you could also have a 'fake' shell that still allows them to gain a lot of details about the inner working of your systems.

>It's pretty trivial to leave SSH on port 22 and just forward a high number port to it while blocking 22 externally. All the root-not-compromised assurances and still maintaining the high pass filter.

Yep! Certainly can.

>While a VPN certainly can be a good solution to securing a machine, you've now got the problem of the VPN server needing its access protected.

There's a few things here:

So, Wireguard uses key based auth, so it's pretty sure by default. Additionally, it doesn't show up in port scans - if you don't send the correct key, you get no response from the wireguard server. You'll get the same 'open|filtered' response from nmap and similar that you would for a UDP port with nothing listening. (I don't think this particularly matters - I don't really care if someone knows I've got WireGuard running - I would run it even if you could tell I was. See following paragraph)

Second, if the VPN is compromised (from a key leak) the worst they get is access to whatever the VPN network is. I don't trust clients just because they're on the VPN, and neither should you - they have all the same auth and security requirements I would put in place for something on the public internet. They still need to get past key auth for SSH (or have a zero day for it). If there's a zero day for wireguard itself, well, now they have access to that box, and can potentially use it for nefarious purposes, but I don't keep any private data for my company or our customers on it, so I replace it and upgrade to a version of wireguard without the exploit. They still need to somehow get past SSH auth, or use another zero day. I think the chances of there being simultaneous public zero days for both at the same time is pretty much nil, and someone being willing to burn two private zero days for the two on you means you're up against someone with enough resources you're probably screwed to begin with.

>Public key only for authentication and strict fail2ban rules combined with port forwarding makes for a very tight system. Not invulnerable but secure enough to not be worth the effort.

Frankly, I think public key only auth is most likely enough for 99.999% of everyone to begin with. I don't think fail2ban or port forwarding from a nonstandard port to 22 matters overmuch, even for filtering logs. If you're going for the "realistically good enough" setup, key only, no root, only have personal nonstandard usernames allowed, that log noise doesn't matter because you can just grep for failed attempts for the usernames you care about and ignore everything else, and the only thing you really have to fear is a OpenSSH 0day, which running on a nonstandard port isn't gonna save you from either. It might buy you a little time to get it patched and not get hit in the first wave, but that's about it.

Someone setting this up well on Linux would be using iptables forwarding, not a user-space service listening and forwarding. Now, this hypothetical attack has to be able to manipulate the kernel's network stack and we might as well stop pretending the low 1024 ports have special significance in that case, either.

If you can crash or modify ip tables then you pretty much have free access into the system anyway. No need to expose a fake ssh host. Just change the ssh binary itself for an infected one..

Well, I've simply changed my SSH port to a different port <1024 (e.g. 950). That invalidates the rest of your argument.

I was confused about that as well. There are plenty of privileged ports you can use.

True but they're still much easier to find in a scan. Most scans will prioritize them.

It's still better than putting it in the same, expected spot. Plus, port scanning is often a red flag behavior (i.e. it's against terms of service in EC2 to perform it from one of their machines), and it can be detected.

It's like putting your key under your door mat vs. some loose brick near your back door. Sure, someone can still find it (I don't recommend the real-life equivalent), but assuming you have a camera and/or nosey neighbors, there's a good chance the presumed invader is going to look suspicious enough to garner unwanted attention.

Oops I didn't know about the scanning thing. I often do this from my hosted VPS to my other systems, to make sure all ports that I plan to have closed are actually closed.

Especially when it involves IPv6 networks and each internal device has its own IP, this can involve a lot of scanning. I haven't had any complaints but good point, I could get banned for that.

It does reduce the noise in the logs. I also use pam_shield to drop packets from the scanners.

>If someone using SSH ignores the fact that the host key has changed

Here's the real problem.

Also if you're worried you could use SELinux, AppArmor, or things like docker to limit the capabilities of a compromised PHP application.

It is a major issue. Things like autoscaling have got a lot of bad habits ingrained in people though.

You should definitely use SELinux or AppArmor. For any production workload I do.

But for any production workload I don't allow SSH over the public internet. I require a VPN connection to a subnet that only has jumphost(s), the jumphost access logs are shipped to an ELK stack, SSH requires both a key + MFA to login to the jumphost, and SSH between production servers is blocked and alarmed on.

SSH onto the prod servers should also be denied. You say autoscaling so do you have an image? If yes then why do you need prod SSH access anyway? If a box is acting up kill it and let the ASG create a new one.

Sometimes you have trouble reproducing an issue outside of prod, even with things like tcpreplay or blkreplay. You could just kill off a problematic instance, but then you have trouble knowing why there was a problem to begin with. Grey failures might not be obvious in logs or metrics.

The idea that you never ever have to SSH into a production server is a nice ideal, but I've never seen it survive reality unless you just shrug about issues occurring and don't mind not being able to root cause them.

> If someone using SSH ignores the fact that the host key has changed

Then it's game over regardless of anything else. That someone is now owned by anything and anyone who cares to MITM them.

Yes, it is an issue for sure.

But it's also a world where people have grown accustomed to doing this - automatic scaling in the cloud with re-use of IPs means that systems come up with the same IP frequently, so people have gotten used to doing this.

For the stuff I manage in the cloud I use their APIs to keep my known_hosts file up to date - if an IP got reused by an instance, I clear the entry in my known_hosts file. If it didn't, I would see the key change alert (verified by modifying the host key on the destination server)

Why not directly setup wireguard on the same machine as the ssh server?

Wireguard doesn't answer anything during port scanning, or when it receive unauthenticated packets, as it doesn't need a handshake.

Actually, I'd think that wireguard + telnet should be good enough, though ssh has a lot more features.

> Wireguard doesn't answer anything during port scanning, or when it receive unauthenticated packets

While I agree that UDP port scanning is harder than TCP, since you cannot just batch send SYNs on every port, it really depends on how you did setup iptables and the sophistication of the port scan.

Most scanners will consider that not receiving an ICMP/port unreachable is a sign that some UDP service is listening. This could be prevented by a default DROP instead of REJECT to confuse scanners, but it has other annoying implications. I bet most people out there do use REJECT so a UDP service such as wireguard would be immediately spotted because of its lack of response.

Also note that if the attacker is willing to invest a bit more time on the map, they will most likely have a Wireguard probe.

Wireguard doesn't respond at all if you do not send the correct key from the start, so I'm not sure how you would write a probe for it. You receive the exact same response as if you send a UDP packet to a port with nothing listening.

Most real world firewall configurations I have seen use DROP instead of REJECT, so anecdotally I'm not sure about your claim.

I don't think it's particularly worrisome if someone knows I'm running Wireguard, though.

If there's a wireguard zero day, they gain no more access to the rest of my servers than if I had port 22 open to the internet - and I don't think that's a significant risk with key based auth to begin with. If there's an openssh zero day, then yeah that could be trouble, but now someone needs to have zero days for both wireguard and openssh to get in to my prod servers?

If they've got two private zero days for that and they're willing to burn them on me, I'm an incredibly high value target and pretty much everything we're discussing here isn't going to be enough to save me, and I've got much bigger problems.

> Wireguard doesn't respond at all if you do not send the correct key from the start, so I'm not sure how you would write a probe for it.

Is that really so? I'm not all that familiar with Wireguard but that seems like a debugging nightmare if the client has no way to get any error pointer from the server. At least my experience from setting up some IPsec infrastructures is that client logs are essential for troubleshooting.

> Most real world firewall configurations I have seen use DROP instead of REJECT, so anecdotally I'm not sure about your claim.

I do advocate for DROP, but most configurations I see are default REJECT. DROP has a bunch of disadvantages that most people don't want to deal with. It messes with TCP because of the lack of ICMP responses, and overall it makes troubleshooting harder because you end up with programs hanging and time outing instead of instantaneous failure.

Actually the use of REJECT is so widespread that it makes my life easier. When multiple levels of firewalls are involved, I can be quite sure that if the program hangs, it's a rule on my side, while a RST tells me that it's somewhere else.

> I don't think it's particularly worrisome if someone knows I'm running Wireguard, though.

Agree, though the subject is just port scanning here, not what happens beyond that.

>Is that really so? I'm not all that familiar with Wireguard but that seems like a debugging nightmare if the client has no way to get any error pointer from the server. At least my experience from setting up some IPsec infrastructures is that client logs are essential for troubleshooting.

Yep! Which, you're right, can make wireguard troubleshooting a pain... but it also mostly 'just works' with significantly less configuration overhead and chance of messing things up than your general IPSec IKE setup. IKEv2 definitely makes things nicer, but it's still not as generally painless as wireguard.

DROPping a UDP packet costs much less that REJECTing it. I cannot imagine why one would use REJECT for those.

I don't know why to mention them in the firewall at all. The kernel knows which ports it is listening on, and drops the rest without prompting.

> DROPping a UDP packet costs much less that REJECTing it. I cannot imagine why one would use REJECT for those.

Because 99.99% of people out there use iptables, with a single default rule for unmatched packets, whether it be UDP or TCP.

> I don't know why to mention them in the firewall at all.

Because you surely don't want your hosts to expose something unexpected to the internet .

> The kernel knows which ports it is listening on, and drops the rest without prompting.

Nope, and that's precisely the subject of the discussion here. In case nothing is listening, you will receive a nice ICMP "port unavailable".

OK, that's news.

Is that the same (externally seen) effect as a REJECT?

A wireguard zero day could in theory result in you getting owned in that situation. It also is just easier to manage if you have multiple servers.

I don't think it's particularly likely, but I don't think OpenSSH server zero days are particularly likely either.

> When you make that port 2222 or whatever, like so many people do, you have cut out a lot of noise... but now that compromised PHP application you had running has allowed someone to now race you every time SSH is restarted for an update or crashed or whatever to bind on that port.

This attack is mitigated quite strongly if you have selinux enforcing and aren't running networked services in unconfined domains.

Even if you have networked services that aren't confined in the default targeted policy, you can probably learn to write policy for them in a day or two, although poor documentation can make for a steep learning curve.

If you're going about that level of security (and you should be!) - why are you bothering with SSH being open to the public internet to begin with?

This is what's weird to me about this whole argument - people can come up with lots of ways to secure this, yet aren't willing to do one of the things that will provide the most security while also offering a high pass filter in blocking SSH access to the public internet. No log noise, no chance of a zero day hitting it, one key being compromised is no longer enough to result in an access breach.

In my case it's because (not counting the machines I get paid to admin, which are indeed behind a VPN) I only admin one isolated VPS, so without a dedicated bastion I feel that the benefit of a VPN is reduced. I just secured it according to the DISA STIG plus some more intrusion detection and stronger selinux confinement.

Adding a dedicated bastion would double my monthly costs, but SELinux costs me nothing if the targeted policy covers my applications, or like half an hour of my time per service if I have to write my own policy modules.

Although, I should point out I'm playing devil's advocate here because my ssh is still on port 22.

I agree, however with any team with size > 1 you then have to manage access. I've seen small teams share a common credential and getting SSO to work OOTB with that VPN infrastructure in SCM is a pain. Additionally, if you end up having more behind that VPN (e.g., db's, monitoring, etc) its arguably net worse to do than the port-shift IMO.

> just set up a VPN on a separate machine and restrict SSH access via firewall to that machine.

So create a tunnel to create a tunnel? Why not port knocking or just white listing static IPs?

Listen on both ports and firewall out 22?

Fully agree. If you want security by obscurity, modify openssh-server and jumble up the protocol a bit. XOR all the bits with some key. Send ssh packets encapsulated in Skype video frames or encoded in the fur of cat pictures over HTTP port 80.

I'm not actually joking -- the above tricks work really well for stable connections through the China firewall.

I think a useful related question is: Does running SSH on IPv6 count as security, or security through obscurity?

Because the public facing SSH services I have on IPv6 have never been probed or had bad login attempts even once in the last four years. I don't consider that a security control at all.

Good question. There is a lot of stuff you can safely move to IPv6 (monitoring, SNMP) and it will completely clean your logs of script kiddies trying get in that way. (Although it does not preclude you from actually securing those things, of course.)

SSH is bit tricky though, because you may find yourself on a laptop in a hotel and then you would first need to create an IPv6 tunnel to be able to get in. If the only time you need to get in is during an emergency that's a hassle you don't need.

If you're going to the trouble of having a tunnel solution then you don't need public SSH or whatever else to begin with :)

It's a pity IPSEC is still such a total dumpster fire on Windows as it works great for ad-hoc with everything else, but thankfully Wireguard has been about as light-touch as VPNs get and I've had good luck getting it out of nearly any network.

>I really hate the accusation that changing services to uncommon ports is security through obscurity.

If you're doing this for security reasons, then it is by definition 'security through obscurity'. The problem is that you don't like the term because it generally carries a negative connotation. What you and the author should be arguing is that 'security through obscurity' shouldn't be seen as a negative necessarily.

>So when you have SSH listening on some uncommon port and see hits in your logs you have a reasonable assurance the attacker isn't some script kiddy spamming the "attack" button on some stupid app.

How do you figure that?

Thank you for this. The reason for moving ssh to a different port is not to stop a determined hacker, it is to lower the risk from script kiddies. It lowers the risk from stupid stuff, and is worth doing.

> I really hate the accusation that changing services to uncommon ports is security through obscurity. It's not a security practice, it's a high pass filter.

Well the high pass filter's purpose is to increase security. Therefore it is literally security through obscurity... Even if it's effective, that's what it is...

No, seems to me that giantrobot's point is that adding randomness to port numbers is a light security through computational complexity (almost a shared secret), that can in turn be used to help prop up other security practices, such as fail2ban. Security through obscurity is reliance in security engineering on design or implementation secrecy

Preventing people from ddosing your auth logs leading to false negative fatigue that lowers the effectiveness of your login monitoring security measures is actual security, not security by obscurity.

Exactly: moving your SSH port isn't security by obscurity because it's not security in the first place.

Wrong.

Anything that reduces the likelihood of success of any attack is security. Non-standard ports essentially remove you from the radar for a lot of dumb IP block scanners. That alone is worth the minimal effort involved.

What you just said it's the exact goal of security by obscurity. If you make something more secure by hiding it, it's security by obscurity.

It won't be inherently more secure, the same flaws will still exist, it will just be less visible to attackers, thus obscurity.

A religious approach is not appropriate for technical issues. What you call "security through obscurity" is not the Devil, to be shunned at every step. That's just a meme that's been uncritically propagated via social media.

Hiding things has its value. Learn how to use all tools at your disposal, wisely.

One could argue all security is "through obscurity". The encryption key exists somewhere in the parameter space, which is open to everyone. You just don't know where it is.

It is more productive to think of the onion model of security. Any technique is just another layer in the onion. As long as you have a sufficient number of layers, and they're all good in different ways, you're probably fine.

> A religious approach is not appropriate for technical issues. What you call "security through obscurity" is not the Devil, to be shunned at every step.

Of course not, and frankly that's a gross mischaracterization of the parent's point. The problem is not that security through obscurity is the devil; the problem is that security through obscurity is not security, and is not useful when you need security.

> The encryption key exists somewhere in the parameter space

This misses the point of cryptography, which is that you (roughly) have to try each possible key separately. So, for example, the combination of a 64-bit security layer (or set of layers) with another 64-bit security layer is a 64-bit security layer, not a 128-bit security layer, which you should know if you know what "parameter space" means.

It is just obscurity.

It's log filtering.

Log filtering has about as much impact on security as having overseas contractors watch your security cameras.

Log filtering at the expense of having one more thing to manage, forget, or otherwise screw-up.

Ways I've seen this go wrong:

- someone fails to copy the port while communicating it

- the new whiz-bang AI security solution detects non-standard SSH ports and "quarantines" you while you try to figure out what happened and who to talk to

- someone manually "reviews" the firewall rules and locks you out of your own boxes

- someone builds a tool that uses SSH but doesn't allow non-standard ports

> the new whiz-bang AI security solution

> someone manually "reviews" the firewall rules

This is a feature. Allowing Unfriendly AIs or incompetent morons to dictate security policy will go horribly wrong eventually; nonstandard SSH ports help make it painful for you to allow them to gain a foothold in the first place.

Turning incompetence into actual problems is not a value-add. I already know the incompetence exists, so there's no informational value, and if I were in a position to do something about it, I would have, so there's no organizational value. Just wasted time.

It clearly makes a difference for me. My servers which have SSH listening on port 22 have countless failed login attempts. Those running it on another random port have almost none.

Me and @p4bl0. I've got one box with ssh on 22. Logs are very noisy and the rules from fail2ban grow and grow.

On the other 20+ boxen with ssh on port xxx22 the logs and f2b rules are much smaller - which means less hassle for the admin.

And on the boxen with services behind WG there is zero noise.

Naturally we're using keys only with all this. The reduced noise in the logs/rules/firewall are very handy.

I love Brian Regan but I have to say, even though I'm probably ruining one of his best jokes [0], that the plural form of "box" is not "boxen".

[0] https://www.youtube.com/watch?v=xkrMsPiqG6M&feature=youtu.be...

Oh, I'm aware it's "boxes". But, I got into the "boxen" habit in the middle 90s and now that I'm "old" I've chosen to become stubborn and stuck in my ways. Also, I think it's more fun. I've got other things to be pedantic about.

I'm pretty sure "Unix boxen" predates Brian Regan's standup.

OpenSSH on port 22 gets at least one attempt a minute for me (and that's with fail2ban blocking IPs on a single failure.) Dropbear on port 2222 gets at most one an hour.

Since I cleaned out and restarted fail2ban yesterday (~1800BST), the sshd:22 filter has banned 291 IPs whilst the dropbear:2222 filter has banned 8.

No, that's the norm.

> Putting SSH on a different port is either done for a specific reason, or [...]

Yes. The specific reason is "it makes log files less full of crap we have to sift through".

Haven't seen one in a while bug a steering wheel club lock had the same effect. A determined attacker could just cut the steering wheel and remove it but there were a bunch of other cars without them.

Non-standard ports greatly reduce the incidence of drive-by shooting attacks. That alone is worth the minimal effort involved.

The argument is that moving ports act on the probability of the attack, not the possibility of the attack.

Reducing the probability of the attack attemp is good in itself and reduces the noise and thus allows easier detection of worse attackers in server logs.

Hell I'm not arguing security by obscurity doesn't have an effect. I do believe it does hide security vulnerabilities that can be attacked, but it's still security by obscurity.

>thus allows easier detection of worse attackers in server logs

I would be amazed if you could show me a single instance where people would see an attacker in server logs and act on it... quickly enough too that it's effective.

I'm pretty sure that it's extremely rare that anyone will look at logs, even small one, to catch theses kinds of situation. If that's really what you suggest as being an effective way to protect yourself, than that's what you should mentions that it's required to do to become effective.

Fail2ban will act noise or not... which was my point in my comment.

So first, moving the SSH port to something non-standard is not the only step in securing a machine. It's a filter.

On my little jump server at home I used to get tens to hundreds of login attempts per hour. The server only allows for pubkey authentication and my fail2ban rules were strict (block after a single failed login) so the probing never went anywhere. Unfortunately the block lists got huge if I set the block time to a meaningfully long period. It also kept my server at a constant low utilization.

I then started forwarding a non-standard port (that didn't end in 22) and the number of attempted logins dropped to a dozen in a week. I could set my f2b rules to expire after days and not see any performance impact from a big iptables blocklist. My server's baseline load also dropped since it wasn't dealing with constant noise. I could also set up alerts on failed login attempts since I wasn't getting inundated with noise.

I started with a secure setup and increased security by changing the SSH port. Less noise means I can actually see when things happen. It's an outlier event when I get a failed login on my server.

> I could set my f2b rules to expire after days

Why would you let the rule be shorter when the attack are more targeted? Doesn't make sense to me to lower it.

> not see any performance impact from a big iptables blocklist.

Wouldn't it be a pretty marginal performance impact? Let say you do get a performance impact significant, then the goal isn't security, it's performance. That's a new thing, performance by obscurity!

> I could also set up alerts on failed login attempts since I wasn't getting inundated with noise.

So you are looking for failed login attempts more now? I have an hard time believing this, but as I said in my previous message, if that was what made it more secure, it's a pretty important step to add, as I would ignore them whether it's 1000 an hour or 10 a week.

I'm curious, what would you even do if you were to see a failed attempt? At what point will you act and how would you act? Anything I can think of is meaningless.

Most distros have f2b's bantime and findtime set to ten minutes by default and a maxretry of 5. Those settings cover the script kiddies/bulk scans. Because of the short timeout your iptables block list isn't going to get too full for too long.

I set my bantime and findtime to much longer durations and maxretries to be 2. This means my iptables block list is long for long periods and f2b has to do more work looking through the SSH logs when dealing with the noise of listening on port 22. Both have non-zero effect on my little Atom powered jump server.

Cutting down the noise let me set even stricter f2b rules with far fewer resources being used on my server.

As for "doing something", there's not much to do for a failed SSH login attempt. I do send myself notification emails but those are just a small section of a larger status email. If I were to see a lot of login attempts on my random high port I'd likely take down external access to my server or set up VPN access to my home network.

My goal is to avoid background noise and get on a "not worth it don't bother" list.

They mean filter out the low value noise (like you would filter out the constant hum of the PSU/grid), the scans on the usual ports (which all happen to be low port numbers).

An FBI agent from the Cyber Crimes division gave a talk while I was in college (>10 years ago). He was interested in brute force attacks against SSH daemons and created a couple hypotheses around number of logins and common passwords. To test this he setup two Honey Pot to record all of the username/passwords. The first one listened on standard SSH port 22, the other listened on a random high-numbered port. He left both of these running for ~6 months.

Results: The honey pot listening on standard port 22 received 1,000s of login attempts (sorry, don't remember the exact number). The honey pot listening on the random high-numbered port received exactly 0.

I know this is just an anecdote and it might not necessarily be true today, but this experiment always sticks in my head. At least the guy used the scientific method: created a hypothesis, conducted the experiment, analyzed his results.

What I've found these days monitoring my own network is that there is now 2 waves -- a port scan and then the attack.

If I change a port for anything to another random port I won't get any login attempts for a few days but eventually I start getting hit again. I can repeat this over and over. I imagine what is happening is that the bad actors are scanning for open ports and they feed that periodically to another process that attempts logins.

The second wave is likely when public port scanning services such as shodan re-scan your host. (I wonder how hard it would be to fingerprint and subsequently blackhole shodan et al's scanning traffic)

Like an IDS or IPS? Snort is quite decent at that.

Anecdotally, I have a machine with an exposed SSH, on a high number port. I get brute force attempts on a regular basis against it, just way less than when I run it on the standard port number. Security by obscurity is just one part of the steps I take with that machine. Using a high port number is dead simple and easily handled client side too, so I just do it.

> just one part of the steps I take with that machine.

You may be interested to know this is called "Defense in depth".

https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%...

The articles summarizes

> It’s where you keep the mechanism secret, not the key.

I think this can be, as you write, defense in depth if the secret of the mechanism is not the only defense.

As example the block cipher for the Common Scrambling Algorithm https://en.wikipedia.org/wiki/Common_Scrambling_Algorithm has been secret. As it seems that has delayed the analysis of the system for about 8 years, but not damaged the procedure.

Technically defense in depth refers to multiple effective security measures (like cryptographic login), so security by obscurity isn't actually part of it.

(Moving SSH port plus something like fail2ban could be considered defense-in-depth against the incidental DDOS-like issues, though.)

Also anecdotally, I've been running SSH on the same 30xxx port since ~2004, including a cluster, which ran a public-facing service that was a popular target for various forms of abuse.

There was not a single brute force attempt. None.

I recently tried to change ssh port to remove log noise. Well, it certainly helped a little bit, but bots quickly found out new port and started to brute force it, so in the end it did not help, just reduced noise. And as I don't see much difference between 100000 attempts and 1000 attempts, I decided to return it back. I don't care about brute force anyway, my passwords are not "root:root".

This made me think of a fun toy-idea.

Let the server and client share a secret. Use that secret to encrypt the UTC date (2020-09-21), and sample some decimals from the first few bits (adding 100 or so, to avoid low-ports).

You could use that mechanism to rotate ports every 24 hours. This way, the bots wouldn't be able to learn the ssh port for more than 24 hours, without the shared secret.

Sounds like fun, or an easy way to lock yourself out of a box by mistake, depending on your perspective. :)

Along similar lines, Port knocking (https://en.wikipedia.org/wiki/Port_knocking) is something out-of-band used to allow access to a host (bypassing a firewall).

I think I'd sooner implement port knocking rather than port-hopping

Ooh, yep. This is definitely a better first line of defense than port-hopping.

Maybe you'd knock to get a number, and hash the number to get the real port.

Defense in depth! :)

I came here looking forward to upvote anyone mentionning port knocking.

This is THE way to go !

You've reinvented HAVE QUICK[0]

[0]https://en.wikipedia.org/wiki/HAVE_QUICK

What is old is new! Seriously, it's cool to see something like this built, especially in the analog (EM spectrum).

I imagine the people who had to use these things locked themselves out all the time. :)

We still use it, and we still have sync issues.

So yes, it's a FANTASTIC way to lock yourself out!

Or use a TOTP with a long period (10 minutes?) and use that value mod, say, 10k with a base of something like 9000. Easy to calculate the port in your head, impossible to guess without knowing the TOTP secret (I think) and can be extended with other fun* rules like "But subtract 10* the first digit" or "Add the first and second digits multiplied".

Ah, this is a great twist! Thank you for this great contribution to the "locked myself out" idea bank.

That's a fair point - e.g. Authy is currently broken on iOS 14 which means I'd be locked out if I was doing this and using Authy (although I have a VPN to one server which can then get to my others.)

Seems like a more effective way would be to use port knocking?

When you have a large base of installations in a big organization, this can make a difference in practice because your incident responders have to sift through less data. This makes much less of a difference when you have great log management and SIEM systems in place. Many places don't, and some hygiene can make a difference at times.

When I see this in practice, the first thing I check is how auth is being done and the overall security of the host. Then, I look for how they are doing SIEM because cleaner logs is a common reason and they'd be better off with a more proactive management approach.

>And as I don't see much difference between 100000 attempts and 1000 attempts

You don't see a difference in a 2-orders-of-magnitude noise reduction when looking for attacks!?

root:root would be crazy, I would only use root:toor5 then I am secure for sure...

/s

//Side note, you should never allow login as root, and you should not really allow remote password login at either, only keys

> root:root would be crazy, I would only use root:toor5 then I am secure for sure...

That's a minimal (read: insufficient) improvement, but if it were behind a real layer of security then it would strictly be an improvement.

> Side note, you should never allow login as root, and you should not really allow remote password login at either, only keys

I'm on board with keys, but why not allow root? If it's secured by a key, what's the harm?

This is my observations as well. For 20+ years I have run ssh on a high port, with exception to my sftp server. The sftp server is hit every day, all day. I have received 0 hits to my ssh port on all my other servers. Even if they hit that port, they would not see anything, as I use a poor-maps port knocking using iptables string matching, but I would still see the attempts in the iptables counters and they are always 0.

FWIW, when I chose my port, I looked at port scanning statistics back in the day, looking for the least scanned ports. It appears those stats have held true for a couple decades at least.

I gave up obscuring years ago, and just use fail2ban.

Now that botnets are brute forcing from thousands of unique ips, fail2ban doesn’t offer much on my servers anymore. I‘ve stopped installing it.

If the attackers are using botnets to distribute the load across IPs, then perhaps we need to distribute the detection across IPs: https://www.abuseipdb.com/fail2ban.html

I used to manage VoIP systems and VoIPBL[0] was amazing.

"VoIPBL is a distributed VoIP blacklist that is aimed to protects against VoIP Fraud and minimizing abuse for network that have publicly accessible PBX's"

It's very similar to what you linked but is targeted to catching VoIP abuse.

[0] https://voipbl.org/

Just curious, what problems does fail2ban suffer with thousands of unique ips? (A crowded iptables I guess...)

I still use it with a super oppressive jail time and few retries, with a few whitelisted IPs and it seems to work ok.

I think the concern is a botnet with n IPs is that fail2ban tracks individual IPs, so if you have any kind of grace period before bannination, they get a linear speedup of n, and if there's an expiration period, get to try n times harder than a single bored script kiddie.

Worse, from an economic perspective, theres enough hosts listening on port 22 that a bot can try instead while they wait for timeouts, so you're not really imposing a cost on them. If you view running a botnet as a form of multi-armed bandit problem, the best you can really do is limit the economic value by slowing them down a tad versus their many, many other options.

F2B support subnets: https://github.com/fail2ban/fail2ban/issues/2261

I think they're saying it doesn't stop brute force attacks because the botnet will just try with another IP.

My logs still show repeated attempts by the same IP. Lots of 1 and 2 tries, but multiple IP's with hundreds or dozens.

Keep forgetting to enable fail2ban.

Fail2Ban is fantastic tool.

But as soon as you found that "PermitRootLogin" can be set with no then all brute forces become useless since they can't match combination of user/password.

fail2ban has other uses: it prevents non-root user error (oops, one of your contractors reused a password…), it significantly reduces log noise, and it protects against any future exploit which doesn’t always work on the first 3 tries.

I know...

But for mine usage it increase memory usage. I'm using it on OrangePi Zero with 256MB RAM.

Port 22 is opened for world so anyone can join. Device have 2 users - root and jacob. I make a change and disable root login from WAN. Now can login from root from LAN. Since noone knows that "jacob" exists i'm saved.

Not necessarily, plenty of people have common / guessable user accounts. For example every one of my servers in the cloud has an account called "user". (All my servers are also key-only authentication, obviously.)

If you use non-standart usernames then you're saved!

Dito, a combination of forbidden root login, an obscure username, fail2ban and disabled password authentication worked well for me for the last 10 years. It's also quite simple to set up. The important part is to double and triple-check each step so that you don't lock yourself out (which has happened to me multiple times in the past, of course).

True, but I still prefer to completely block IPs who tried to log in via SSH and failed multiple times.

Sure, but enforcing best practices at all times can be tedious and unnecessary sometimes.

You may have summoned tptacek with that.

(He would argue to turn off passwords and use keys only)

IMO fail2ban is still useful just to cut down on the noise.

Yeah, I still support servers that have (strong) passwords without keys. I use fail2ban on them.

I use source tracking rules on pf protecting my servers and firewall itself:

https://www.openbsd.org/faq/pf/filter.html#stateopts

I get thousands of tries in a day to a newly installed host using IPV4 nowadays.

The entire point of everyone who rants against non standard ssh ports is that it adds more noise to the signal of "You should disable password login entirely and only use public / private key authentication."

And reading many of the responses in this thread it makes sense to me. So many people in this thread talking about how many less failed password login attempts they get when they change the port, indicating they allow password based login in the first place.

Even with key auth only, which I have, you're still getting many attempts, wasting CPU cycles.

Most scans are smart enough to give up as soon as they see publickey as the only method, but some keep trying to connect hundreds of times.

If you expect to be hit by that kind of attack (simple combination of username/password), then you should protect yourself from that kind of attack. It's never been easier nowadays to do this.

You may answer that you could still miss a few of theses simple password, that your solution would be more effective, sure, but then you use security by obscurity to protect yourself.

By the way, security by obscurity does works, it's not bad per say, just as that FBI agent just proved, it does have an effect. If it didn't, there wouldn't be so many case where it was used. The issue with security by obscurity is when you rely on that to protect from vulnerabilities and then ignore them. It only lower the likeliness of getting attack, it doesn't make attack less effective, it doesn't protect from any vulnerabilities.

Sadly, too many time, we just ignore it, hide everything and hope to avoid targeted attack which would foil that obscurity pretty quickly. This is when it get bad.

I have the same experience with non-standard port usage, and I think it's a very reasonable thing to do, while also caring for the security of the service behind that socket. SecOps will thank you for not having to wade through log spam in the endeavor of preventing attacks.

By the way, does anyone here know how to actually check sshd logs on MacOS?

There are a number of responses here: https://stackoverflow.com/questions/43382825/where-to-find-s...

Don’t disable port 22; just neuter it. If port 22 is disabled, the attacker will look elsewhere in your system.

This is an interesting point. Imagine if you put a fake SSH agent on 22, it responds just like SSH but never allows a login. Would it make it even less likely that someone would bother trying another port?

Obviously the next step here is a fake SSH agent that allows logins to a little sandbox.

Probably a bad idea, one vulnerable service invites people to look for more.

Depends what we mean by sandbox. I wouldn't make a chroot the honeypot, but I don't see an issue with a program that just simulates a shell but doesn't allow exec or real fs access, for instance.

I'd be weary of that. Sandboxes have been escaped.

I don't think you would ever let them touch OS-level resources. There are plenty of third-party ssh server libraries where you just get a Reader and a Writer to the remote end. When they connect, you write "root@cool-computer# ". When they send bytes, you discard them, then print "root@cool-computer# " again.

While obviously accepting a TCP connection and allocating resources on your computer is more risky than just ignoring the connection, presumably it would be fun to do this, which is a good reason for doing something. You can set a memory limit, file descriptor limit, etc. and just crash if they're exceeded. You can run your little fake ssh daemon in gvisor and protect against attacks nobody even knows about yet. All in all, it would be pretty low risk, and also pretty interesting.

And the next beyond that is a sandbox that contains simulated data....

honeypots are obvious.

They won't and that's the point of the OP.

"They" are just a bunch of bots that try to hit 22 once and that's it. If it's open report it's open if not then move onto the next target.

The only time a human is going to look for SSH on non-22 ports is when they are actively attacking you or your organization.

Do you have a suggestion on how to do this?

An SSH Tarpit is a good way:

https://nullprogram.com/blog/2019/03/22/

thanks!

It depends, many people do not do a full NMAP scan. Takes to much time.

Then again, is port knocking security by obscurity enough?

Sometimes I run an extra http listener on 22

Do you know if he ran the honey pots on the same server? if so that's why it received 0. the bots hit 22 and stopped.

Different servers.

> It’s an useless anecdote because SSH bruteforce attempts are not a threat and cost you nothing.

I can say from personal experience that this anecdote is both accurate (20 years ago and up till today) and meaningful.

No idea where you get this notion that brute force attack are no threat or without cost.

They certainly do pose a security risk (takes only one insufficiently trained employee/intern for a potential breach) and they certainly come at a cost (way beyond dirty just logs).

Botnets and fast networking stacks like DPDK have made port scanning the entire Internet a much more viable proposition than 20 years ago. Depending on your sshd settings you can be effectively locked out of your machine by a brute-force attack. Running on IPv6 and/or having a secondary sshd instance that only accepts connections from whitelisted IPs is cheap insurance.

That doesn't invalidate the observation (which I share) that these attempts are almost 0 when using a different port. It reduces logspam and if I start getting lots of brute force attempts on my non-standard port, this is useful and meaningful information (someone cares enough to do this).

> Botnets and fast networking stacks like DPDK have made port scanning the entire Internet a much more viable proposition than 20 years ago

True indeed, yet even today I have seen little evidence of scanning beyond standard ports (pretty much the same as in the past). Criminals are opportunistic by default and tend to go for low hanging fruit (standard ports, with standard server config). I certainly did see in increase on standard ports. Even while full range scanning has become more feasible, I have not seen much evidence of its use.

> takes only one insufficiently trained employee/intern for a potential breach

How? If they've leaked their key, why do you assume the port hasn't leaked too? On the other hand if they haven't leaked their key how would they get in?

Or are you allowing password authentication like it's 1999?

> Or are you allowing password authentication like it's 1999?

That is assuming you have such authority or technical means. If you're maintaining systems for a company, there's a good change that the product vendor simply won't allow fucking around with their system like that (ergo: yes, in practice you are indeed stuck with your 1999 authentication).

I'm not saying that it is good security (that's why layers security is often paramount), but it is situation I've encountered more than a few times.

Great for you, if you are GOD on all the systems you work with. Even then, your client/employer might simply tell you to stuff your objections and accept the bad authentication policy, because to them the risks are simply not worth the business disruption. I totally agree that is a flawed argument. But decisions usually aren't always (if ever) called on valid arguments.

Good for you, if you are in a position where you never had to deal with such real life situations.

Heck, some cloud providers have password logins by default. Since the instances are easy to setup I'd imagine many companies operating with a no-ops situation are vulnerable and don't even know it.

And then there are side projects. I remember being educated enough to know better, but doing it anyway as the server was a $5 digital ocean droplet, used to run a tiny minecraft server for some friends. Got brute forced and spent the next two weeks red-faced, trying to get DO to allow network access again so I could at least grab a backup before nerfing the droplet.

Now I use a basic ansible setup to automate changes to sshd so I don't have any excuse to be stupid again.

Not sure what you're arguing here. You either have control over sshd or you don't. Or are you really suggesting you can change the port of sshd but aren't allowed to disable password auth?

I'm a software engineer, so if my company gets hacked via ssh that's really not my problem. Worrying about such things would make me a busybody. But if you're a system admin and can't properly do your job, then I would seriously start looking for a new place to work. They will get hacked and you will be the guy that gets blamed.

> Not sure what you're arguing here. You either have control over sshd or you don't. Or are you really suggesting you can change the port of sshd but aren't allowed to disable password auth?

First, you'll have to separate two things here. One is the technical ability to control sshd, the second whether a company will allow you to tinker with the auth policy (whether that is password login, password login with only strong passwords, or rsa/ecdsa key access only).

The latter has nothing to do with control and only with what decision makers allow you to do (that sometimes is a large product vendor, not allowing anything beyond what they ship). If you work in a place where you have full control over the systems you work on, great for you. I can ensure you that it is not the norm (unless we're talking about hobby projects or projects with exclusive personal ownership).

As for the technical aspect, keep in mind that changing the public facing ssh port might not even be done on the host itself, but e.g. in port forwarding table in a router/firewall. This might not even always happen because it's technically impossible to do it on the box itself.

I'm pretty certain that tinkering with a box is regularly discouraged (especially if it is managed by some orchestration or vendor specific control/update tool), while effectively the same can be done by changing a router/firewall. There's a lot more things to be said about that, but please take it from me that hacking around in a systems you have not build yourself isn't always a bright idea (and it happens to be a very common situation).

> But if you're a system admin and can't properly do your job, then I would seriously start looking for a new place to work.

That's an interesting theory, but frankly not how I think the real world (usually) works. As a system admin you are there to solve problems for a client or employer. You can (and should) of course always warn for potential dangers, but refusing work or quitting a job/assingment because you're not getting full control over a system .. good luck with that. It is simply not an acceptable position in many situation. You must be in really high demand if you want to pull stunts like those and still have any work after a while.

Maybe it works different in software engineering land, but I highly doubt it. When was the last time you quit a job, because you preferred a different library or framework over the one your superiors/client dictated?

Please don't get me wrong. On a personal level I'm very principled about what I choose to work on or with (and what I refuse to take part of). But at the end of the day we are professionals, here to solve problems. If we can and a client/employers is willing to accept the risks of an imperfect solution that fits in their requirements, it ultimately is their call and responsibility. All within reason, of course.

> tinker with the auth policy (whether that is password login, password login with only strong passwords, or rsa/ecdsa key access only).

Or the port ssh works on

> that sometimes is a large product vendor, not allowing anything beyond what they ship

Surely you'd limit access to that on an IP level and bounce via a bastion (which you do control)

> tinkering with a box is regularly discouraged (especially if it is managed by some orchestration or vendor specific control/update tool), while effectively the same can be done by changing a router/firewall

"Tinkering" with a router/firewall sounds far more dangerous than a box -- you can knock out 2000 machines in one go.

> That's an interesting theory, but frankly not how I think the real world (usually) works

If the shit hits the fan, are you confident your management (which apparently refuse to allow you to implement basic security policies) will have your back, or will they pile the entire blame on your to save their skin.

Better to look for a company that respects your skills before you get pink slipped.

> Or the port ssh works on

Calling that part of the auth policy (in the context if was responding to) is a bit of a stretch, but okay.

> Surely you'd limit access to that on an IP level and bounce via a bastion (which you do control).

What percentage of organizations have you seen do it that way? In my experience it's more often directly behind an internet facing NAT router, through a port forward. I'm not saying that's a good thing, I'm saying it's reality.

> "Tinkering" with a router/firewall sounds far more dangerous than a box -- you can knock out 2000 machines in one go.

You again appear to be missing the point I tried to make. It's not so much about danger, but more about control. A box is regularly far more of black hole (especially if it's a vendor appliance or legacy system) than a company's router/firewall is. Sure not without dangers, but that's why you're a professional that (hopefully) knows what he/she is doing. How often did you work on a router/firewall that controlled 2000 machines? In my case, I can count those on one or maybe two hands.

> If the shit hits the fan, are you confident your management (which apparently refuse to allow you to implement basic security policies) will have your back, or will they pile the entire blame on your to save their skin.

It works a bit different if you're contracted or working for clients, but either way: that's why you document things and make clear to those who make decisions that the risks are theirs and not yours.

--

But seriously though .. I'm not sure if you genuinely missed the point(s) I tried to make, if you might be pedantic on purpose (just for the sake of it), if you might be just another armchair general, or maybe have only worked in very privileged positions where you had full control and authority over the systems you had to deal with. The latter is certainly not the reality I've experienced for over two decades.

Maybe you are experienced, just in a very different reality/industry than mine. Still, I find these kind of arguments about companies "not allowing you to do basic security" or "not respecting your skills" rather childish and out of touch with the reality. I have not seen many gigs/companies where sysadmins (or even -architects) have this kind of god-like status. When I did see such situations, it often meant a company would have serious (potential) issues if/when their "guru" would piss off (leaving a collection of equipment in "status unknown", i.e. the next guy would not be allowed to touch anything and ergo my point about tinkering with boxes being discouraged).

How long have you been doing this (professionally)? That's not a rhetorical question. I'm genuinely curious.

Network administration (and system administration of about 150 linux machines) for about 10 years. I did a full port sweep of my network a few days ago, 1,555 IPs with port 80 open (although to be fair several of those are multi-IP connected). Before that 7 years of system administration and development.

My (my team's) network policy is that those web ports are not exposed on the internet - we provide proxies with 2 factor authentication up front. We find we get far happier users when we use carrots.

We operate a high wall policy, and while we do push towards a secure-everywhere system, we are more flexible that other corporate networks, and tend not to have exacting requirements. Your black box wants to use SNMP v2? Of course it does, that's fine. No you're not probing it from the internet though, we'll work with you to increase security.

If a team want a device that claims to run a proprietary protocol and needs TCP ports from the internet, that's fine, we do it. We discovered recently one of these devices was actually running a standard webserver on one of these ports after a firmware update. The device user didn't even know.

Ultimately we provide a network, you can take it or leave it, there's competition (go for one of the two non-shadow IT networks, or build your own).

I know from personal experience what happens when things go bad against my teams advice, all those emails saying "this will go bad" are work jack squat. Fortunately we had very air support for that specific event (front page national news), nobody cares about "I told you so".

Fair enough. Clearly not a greenhorn.

From the context of your description, your earlier arguments certainly make more sense, more than they would in the situations I'm personally more familiar with.

My main conclusion here is that we are both referring to very different (work) environments.

For what it's worth, given the context you've described, I can well understand and support your arguments.

That said, I'm not so sure your context is representative of the industry as a whole. To be fair, I really don't know what would be. Only that we apparently have worked in very different environments with very different conditions and requirements.

Just to be clear: when I argued about warning for the risks of particular choices, I was never referring to internal company communications. Those are indeed worth little to nothing (if shit happens). I was referring to communication between separate legal entities. Withing B2B contract work, such communications do quickly become crucial (if shit happens), even from a legal perspective.

I hope you can see that while your arguments do hold well within the world you know, they might not be so applicable or useful in other (certainly existing) situations. The world certainly is more diverse than the context you've given.

Kind regards, and thank you for answering my question.

> Withing B2B contract work, such communications do quickly become crucial (if shit happens), even from a legal perspective.

Hah. Depends how large the companies are, but again in my experience all that B2B stuff is meaningless. Maybe it's only my company that's terrible at writing, measuring and enforcing service levels, and certainly awful at extracting any penalties (I guess because any downtime doesn't have a direct monetary loss, just a reputation lost which eventually leads to monetary loss)

That said my entire industry (broadcast) relies massively on IT - far more than in the past - and has absolutely no clue about security. In 20 minutes I found 130 of 200 devices on the internet with default credentials open on port 80. Case in point, using shodon, I can find a server and see within seconds that a Polish broadcaster is currently streaming some people playing violins from a studio - not sure if this is a live TV broadcast or being taped for later, it might be going out on "Program 2" on Polskie Radio, but I'm not an expert in the Polish broadcast landscape.

I'm just amazed how anyone could be in a position to have knowledge and authority to change the port SSH is listening on (thus breaking peoples workflows), but not change away from using passwords, even if a bastion and/or ip whilelisting isn't allowed.

They cause some real work because of the log noise they create. It's easier to see targeted SSH attacks if all the undirected attacks are filtered away.

This is absolutely true. I use fail2ban and I often find that it's using rather more CPU than I'd like. Sounds like moving my SSH port might solve that!

>I use fail2ban

Why? Unnecessary attack surface that provides no tangible benefit.

Sketchy parsers operating on untrusted, unstructured log data. fail2ban is worse than useless

What would be indications of "targeted" ssh attacks, and what can you do in response?

If you get 10,000 attempts on port 22, you're probably connected to the internet. If you get 10,000 attempts on port 63290, someone has taken a specific interest in you.

What are you going to do with this knowledge?

Personally? I'd decide the utility of having it public-facing is no longer worth the risk, and firewall it down to a much narrower set of source networks. I'd probably take a moment to brush up on my key hygiene too.

Maybe you should do these things before you realize that someone is trying to hack you?

The fact that someone bothered to scan the entire range (or find your port at random) might indicate that they're specifically targeting you, and just being aware of that is an upside.

>and just being aware of that is an upside.

but what can you realistically do with that knowledge?

Infinitely more than you can do without that knowledge

Give some examples then. Knowing that someone is targetting you shouldn’t change anything, you should be ready regardless.

It shouldn't, but it does. Many smaller companies driven by business people, where maybe tech is just seen as a necessity on the side need a narrative like "people are trying to get in and if they do it's going to be a disaster" to take security seriously. Then or at the point where the disaster strikes.

I'm not really sure why this point was voted down below either; just because you work for someone who takes security seriously (at least to the point where it's insurance-satisfyingly safe) does not mean everyone does.

Years ago I worked at a small agency and every bit of time I spent had to be justified and produce tangible/visible results. "But is anyone really going to try to hack this local business" was a question I actually had to answer, since most other employees were creatives.

Argue for more security budget with management. If you can make this sound dramatic enough...

They fill the logs with noise. That's "something." I finally set up fail2ban a while back... it works wonders.

Do people even read the logs? What value does it add compared to just monitoring successful logins?

There's a lot of value. For example, if you see failed logins against random user names like "dbadmin" or "root" it's likely just random scanning, but what if suddenly lots and lots of valid user names appear?

That's a great point, but I get back to the root question: who's actually looking at this? If people are examining logs it's usually for a particular trigger or a problem and filtering that signal from the noise is hard.

Likely, nobody is directly looking at the logs.

But they might be using software that automatically raises an alert when it sees repeated login attempts for a valid username.

Isn't that one of the purposes of Splunk?

It's more typical of the servers-as-pets than servers-as-cattle scenario, but sometimes one is simply curious [or extra cautious]. SSH honeypots exist at least in part for this reason.

> servers-as-pets This is a great way to put it.

> who's actually looking at this?

Well, your security team, post incident. But also automated systems like fail2ban.

And log-collectors like Splunk (with configured alerts, etc)

grep and zgrep will work wonders for checking for actual usernames in these logs even if they have significant amount of spam in them.

>but what if suddenly lots and lots of valid user names appear?

then what are you going to do?

Well that would highly depend on what I'm seeing. If it's a single user there might be an attack on the way against that user. If it's multiple users, there might have been a compromise of some credentials.

It's definitely something you need to investigate.

At a minimum, spend some of my limited time and attention on this issue rather than the 100s of other things that might be clamoring for my time.

What are you going to do to solve that issue?

Did you ever had the "pleasure" of a server grinding to a halt because the logs filled up all the space? To where you had to mount the disk to another system and clean it up before it wants to boot from again. Can be a bitch if it's a machine on a remote location. Not everything is cloud (yet) these days.

Granted, there usually is a lot more at fault when you run into such problems, but I find people not looking at logs a rather weak argument for letting them get spammed full with garbage. Certainly terrible hygiene, at least.

>Did you ever had the "pleasure" of a server grinding to a halt because the logs filled up all the space?

I've never seen this issue on any systems I manage, mostly because they all have log rotation.

>but I find people not looking at logs a rather weak argument for letting them get spammed full with garbage. Certainly terrible hygiene, at least.

Why is it weak argument? If it's something that doesn't materially impact you, why should you expend effort into remediating it? Hygiene is only important for things we interact with on a regular basis. We as a society don't care about the hygiene of the sewer system, for instance.

>I've never seen this issue on any systems I manage, mostly because they all have log rotation.

Ah, yes - the age-old claim that log rotation will magically stop a belligerent from dumping 100s of gigs of log files before `logrotate` has time to run ... filling up your disk

And even if logrotate did try to run, you have no space for the compressing file to live while it's being made

What fantasy world do you live in?

a system that grinds to a halt because of overflowing logfiles is already faulty.

What fantasy world do you live in?

Yep, if the sewer system occasionally spills into a river, you can just put up a sign and ignore it.

>Did you ever had the "pleasure" of a server grinding to a halt because the logs filled up all the space?

Everyone should be using logrotate, and if they actually read the things, shipping logs to ELK or Splunk or Greylog or whatever.

> Everyone should be using logrotate, and if they actually read the things, shipping logs to ELK or Splunk or Greylog or whatever.

Certainly they should. That is, if they have that much control over the server and if it's not some legacy system build by some defunct organization or John Doe. I do not disagree with your theory, on the contrary. But then there is reality, where this theory isn't always feasible.

>Everyone should be using logrotate

Doesn't run often enough to counter belligerents dumping 100s of gigs into logs too fast for logrotate to keep up

> Keeping the log file on the boot partition was the first mistake

Wrong assumption. With logs on a full system (but not) disk, your system can still grind to a halt during boot. Sure, if you do have access to the bootloader, you can do an emergency/recovery boot. But you do not always have that on systems build by others (especially product vendors).

I would not be making this point if I had not run into situations where this was an actual problem. I can assure you it was never the result of my personal bad architecture or maintenance and almost exclusively while dealing with third party products.

It would be valid to argue they should get their shit together, but the reality is that at the end of the day, companies buy systems like these and you still will have to deal with them.

Wish I could give you extra upvotes - I keep logs on a separate partition/drive

Doesn't mean the system cannot crash (because logs can no longer be stored)

This article engages in quite a bit of mental gymnastics to avoid calling this tactic exactly what it is. Security by obscurity

What are you doing here? SSH generally runs on port 22. You move it to an alternate port to thwart a naive intruder's attempt to establish a connection to a service running on that port.

You are still running SSH with the same old setup. If it was insecure before it remains insecure. The only thing you've done is "obscure" how to connect to it by using an uncommonly used port.

Why all the hate on "Security by Obscurity" It's a perfectly valid thing to do. It will definitely defend against a host of intrusion attempts. Its only cast in a negative light when it's the ONLY thing you do or if you consider it sufficient in terms of security.

FWIW the key analogy is broken. Hiding a key to your house in a plant is a form of "Security by Obscurity". Anyone that finds the key can open the door. Thats not very secure. So we hide the key so no only people that know where it is can find it. If an intruder found the key they'd immediately have access to the house. We rely on the obscurity of its location to provide whatever security it has to offer.

You could achieve the same affect by having a hidden door and leaving the key out in the open or hiding both. What use is the key if the location of the door it opens is "obscure". This works because access is defined by knowing the location of the key and the door simultaneously.

Both are examples of "Security by Obscurity"

I think you're confused a bit here.

The mechanism is known for a door lock. It's a lock with a key. What they don't have is the key.

If you leave your door key, or RSA key, lying around where it's easy to find, well...that's a separate problem.

So... Are you saying that moving your ssh port to a different address doesn't actually increase your security at all? In my practice it gets rid of ~95% of probes from people trying to break into my system. Whether it just signals that I know what I'm doing, or demands resources that the attackers aren't willing to give, I don't know, but it definitely has a positive effect on the security of my system.

>So... Are you saying that moving your ssh port to a different address doesn't actually increase your security at all

No I think if you read particularly the "Why all the hate..." paragraph you'll see I'm saying the opposite of that. There is nothing wrong with obscurity but in terms of security its the icing not the cake.

By that I mean it will make a secure setup more secure, but its not enough on its own.

>So... Are you saying that moving your ssh port to a different address doesn't actually increase your security at all?

Having a quagmire next to your castle walls don't makes your walls harder to breach, just buys you more time until the enemy gets there.

It's not bad thing to have, but ... just don't build your walls lower and thinner next to the swamp.

Aren't most SSH probes trying to break weak passwords? If so, having a strong password is much much more important than the port SSH is on.

Really, you should not be using passwords at all. We have key-based authentication for a reason.

Totally agree.

Walk and chew gum.

He's saying that it doesn't make your SSH configuration or SSH itself any more secure.

Would banning any IP that tries connecting on 22 increase security?

You could do something like that, but you'd have to write it yourself. If you want to ban IP addresses, fail2ban is probably an easier starting point. Plus users who authenticate with passwords might occasionally make typos and get banned, incentivizing better logon techniques.

For my random high numbered IP ssh port, I see 5-15 attempts per day. In the last 3 months, the most failed attempts from a single IP is 3. Most attackers (at least amongst those that try higher ports) are distributed these days.

how do you connect rightfully then? at that point is easier to have and access rule on the firewall and only accept connections from the allowed ips.

And in this case you are putting the lock in a place that is not the doorknob, where an intruder will typically try to unlock the door.

Albeit a now invisible knob that must be searched for via a door scan. While I don’t personally switch my ssh port from default, I would imagine doing so would cut down on a ton of the noise from lowest common denominator ssh attempts.

I have changed some default ports and seeing the number of connection attempts drop from <many> down to close to zero was just nice. There are just fewer connections opened, fewer logs to analyze and so on. In my view, fewer connection attempts means less data to analyze which in my view is a security and cost benefit.

My entire house nearly instantly becomes invisible to anyone doing a door scan. This is what an effective psad/fail2ban setup can do.

The problem is when you only use security by obscurity. Just changing then port does not do enough to make your insecure application secure. For example, if you have a Windows 2000 server on the internet, it doesn't matter what port you change IIS to, it's going to get hacked.

To use your analogy, if your lock is broken, or can be broken, it doesn't matter where you hide the key.

> The problem is when you only use security by obscurity.

He literally says this exact thing in his comment.

I'm agreeing with him.

You're missing the point; the point is that it's only bad when it's used instead of (say) disabling SSH if it's not needed, or using the latest release, or using secure ciphers.

Doing all of the above as well as changing your sshd port will only improve your posture.

What makes this even more ridiculous is by the most literal definition, most of modern cryptography is "security by obscurity." My TLS or SSH connection remains secure because my selected private key remains obscure and hard to find in a space of 2^128 possible keys. Only a handful of techniques (quantum key exchange, one-time pads) are not relying on obscurity for their security.

Security through obscurity works perfectly well so long as you correctly quantify the size of the search space, the bits of information provided as hints, and the cost of searching--which basically describes what much of modern cryptography is about, at the primitive construction layer. By the same measure, a multi-million dollar cache in the American Rocky Mountains remained unfound for a decade despite there being a poem and short story filled with clues as to its location[1]. If I go totally off-grid for 6 months, and then come out of the wilderness saying that at some point during that wandering I cached my valuable possessions in a secret location, you're not going to find it.

The admonition not to use security by obscurity is well-meaning but requires explanation. We shouldn't hide things in ad-hoc search spaces that haven't been adequately formalized. We should prefer mathematical puzzles which have well characterized properties. And generally that is good advice. But that doesn't mean we shouldn't also throw in whatever extra protections are convenient, even if they're ad-hoc or provide insufficient bits of security by themselves.

[1] https://en.wikipedia.org/wiki/Fenn_treasure

> Its only cast in a negative light when it's the ONLY thing you do or if you consider it sufficient in terms of security.

Yes, that was the takeaway from the article.

Yes the attack surface before and after is the same ergo this is not security at all. The difference between 22 and 2222 is a nominal delay on a port scan.

It would be far easier and more convenient to use a ip whitelist and a firewall to protect access to 22.

That's incredibly tenuous.

Also I don't think you know much about asymmetric encryption or the SSH protocol either.

So if I have your private key and your hosts definition in your ssh config, you're still secure?

Cool, send them to me and I'll show you otherwise. My email address is in my profile.

The point was that nuance is important. Keyspace size, the cost and penalties of a guess, if any information leaks in the process, these are important.

These cutesy memorable rhyming platitudes in any discipline are usually either meaningless or counterproductive.

I could for instance, use a very long base64 url endpoint as effectively a shared password. If you look at the cost, leakage, and keyspace analysis you can tell why it works or it doesn't far more than the old obscurity aphorism.

I think some of computer security theory could be taught via physical lockpicks pretty well so long as the students actually do the physical lockpicking act. I think the concepts translate well to the material reality especially if you have a large diversity of locks.

It actually sounds like a pretty brilliant educational approach.

Computer security through lockpicking. Requirements: introductory number theory, discrete mathematics, 2 or more programming classes

Please don't post like this, i.e. just putting someone else down. It's not in the spirit of the site and doesn't add any information. If you know more, that's great, but in that case please share some of what you know, so we all can learn.

https://news.ycombinator.com/newsguidelines.html

Moving your SSH port isn't really about security. It's about reducing noise in your logs from annoying port scanners constantly hitting common ports. Although that does then have the benefit of making your logs more useful for detecting actual attacks. And it probably reduces some CPU cycles too as an added bonus.

I've had the SSH server for my host on a non-standard port for well over a decade now. My observations:

Back in the mid-2000's, botnets were small, the Internet was slower, and there was a lot of low-hanging fruit. Putting your SSH server on a non-standard port was a good way of reducing log clutter.

Today, though, it doesn't matter. I get as many login attempts on my non-standard ports as I see on hosts running SSH on port 22. This is because for at least the past few years, it has been feasible and indeed entirely commonplace to scan the whole IPv4 internet, 65k ports and all. Companies like Shodan and Censys sell access to detailed maps of basically everything on the routable Internet. There is no such thing as "hiding" a service on some little-used port anymore.

One can make the argument that attack surface is inversely proportional to security, so putting services on non-standard ports is in fact an increase in security. It's just that even back in the day, that increase was too small to seriously consider and today it's microscopic.

> Today, though, it doesn't matter. I get as many login attempts on my non-standard ports

That is weird, I dont see any login attempts on my non standard ports on my multiple servers.

People really lose their mind over changing the SSH port. It's not an either/or choice between a non-standard port and all of the other SSH security settings. Nearly anybody that bothers to change the port is also going to follow most of the other industry-standard practices.

It's pretty much one of the first things I do when setting up a system... ssh port, no password logins, ufw. Of course, when I mess one of those up, I'm doing it again... but in general, that's my getting started step.

If it's a really important system I'll go a step farther and enable a port knocking sequence... but that's about the extent of it. Generally speaking, I rarely setup port knocking.

Same here. Just changing the port makes brute force attacks so rare that I feel I don't need to bother with port knocking or the like.

65022 for the win... it's funny but many scanning scripts stop at 60000 or 65000, or only target specific known ports.

One thing to keep in mind here is that there is no absolute security. There is only security against some specific threat.

If the threat is "someone is specifically targeting you", then changing the port is indeed useless. However, if the threat is "a botnet trying to abuse a new openSSH vulnerability", then changing the port does help. Because no bot takes the time to scan tens of thousands of ports in that case when there are millions of other hosts around who listen on port 22.

By the way, when I lived in a dormitory, it was actually best practice to hide your valuables from view if you lived in a ground floor apartment. Literally obscuring the thing, but thieves only broke into the apartment where they knew there's something to steal quickly. A napkin over your wallet might not help in a bar but it did help in this case.

> However, if the threat is "a botnet trying to abuse a new openSSH vulnerability", then changing the port does help. Because no bot takes the time to scan tens of thousands of ports in that case when there are millions of other hosts around who listen on port 22.

It takes about 8 minutes to scan the whole internet for port 22. If there's an SSH 0day, and I have a botnet, I am going to make the entire botnet scan the entire internet on every port, because I know people move their SSH port. With 5,000 hosts, I can scan all 65,535 TCP ports across the entire internet in 104 minutes.

How fast can you patch your server after the 0day's been announced? What if you're asleep?

The time it takes to change the SSH port could also be used to harden the sshd config, which would eliminate several classes of previous exploits. A little more time and the entire system could be hardened, so that even if you are attacked, the attacker can't do much of anything.

Have you done much with IPv6 scanning?

I know some researchers have set up public services like NTP servers to try and build a catalog of active addresses but with the privacy address extensions even that seems functionally infeasible to me. But most of the things I care about are on private networks these days so I haven't had much cause to seriously investigate.

I haven't; there are definitely some protocol scanning differences. But off the top of my head, you could try 1) do a reverse lookup on a v4 address and then query the name for a v6 address, 2) do DNS probing for a domain you're interested in, 3) try the v6-in-v4 mapping to find v6 addresses mapped to v4 addresses. If you could sit on a backbone and sniff DNS traffic or something, that'd be a nice sized cache of addresses.

> Literally obscuring the thing, but thieves only broke into the apartment where they knew there's something to steal quickly. A napkin over your wallet might not help in a bar but it did help in this case.

The same approach works well for leaving stuff in cars. Visible bags or electronics tend to get stolen. The same thing under a sweatshirt doesn't.