The most reasonable "update" to FTP would be to formally replace it with HTTP, because FTP is an awful protocol --- maybe the worst one in common use --- that deserved to die off decades ago.
FTP includes support for record-oriented files (STRU R, MODE B). This mostly isn't supported by FTP clients/servers on Unix-like platforms or Windows, but it is on those platforms which have record-oriented filesystem support (IBM mainframes, IBM i, Unisys mainframes, OpenVMS RMS, etc.) Although one could standardise a mechanism for transferring record-oriented files over HTTP, no such standard has been widely adopted. If someone wants to transfer a record-oriented file from e.g. VMS to z/OS and have the record boundaries kept (and with the necessary ASCII-EBCDIC conversion applied), FTP is the only widely adopted standard that can do that.
This is also why these platforms often use FTPS (FTP over TLS) instead of SFTP (SSH-based) – SFTP doesn't include any support for record-oriented files, only the stream-oriented files used on Unix and Windows.
> Although one could standardise a mechanism for transferring record-oriented files over HTTP
Or you could just provide a simple API over HTTP, whether an actual REST system or a single endpoint one or two params with well defined inputs that can be accepted (a CGI, basically). Why bother formalizing some standard when the tools to handle this are so ubiquitous (Apache+$LANG on the server, cURL or wget on the client)?
> SFTP doesn't include any support for record-oriented files, only the stream-oriented files used on Unix and Windows.
That's because SFTP isn't really FTP (in the protocol sense) at all. It's just a specialized shell started after an SSH session/tunnel is created.
That it includes FTP in the name is really just marketing because they wanted to supersede the real FTP. In that respect, it makes sense for them to just cover what 99% of the users of FTP needed and stop.
curl and wget on the client can't easily do this, when the client is another mainframe/minicomputer OS with a record-oriented filesystem. I don't believe they have any platform-specific code to support record-oriented files. You probably can get it to work with external configuration (e.g. on z/OS, using JCL to invoke curl/wget with a DD statement which sets the necessary dataset parameters.) But FTP-over-TLS is already a well-documented and well-understood technology in mainframe environments. What possible advantage could one get by replacing it with something hacked together with Apache/curl/wget?
> I don't believe they have any platform-specific code to support record-oriented files.
Yes, I was assuming a more traditional client, which is probably a mistake on my part.
> What possible advantage could one get by replacing it with something hacked together with Apache/curl/wget?
Well, there's always the obvious one, which is firewalls are much easier to deal with, since there aren't two separate ports in use, so you get rid of a whole class of network and firewall errors that are quite common.
> Well, there's always the obvious one, which is firewalls are much easier to deal with, since there aren't two separate ports in use, so you get rid of a whole class of network and firewall errors that are quite common.
If possible, one should use extended passive mode (EPSV) over TLS. Then, we are just talking about two ports instead of one, without any of the connection tracking complexity on middle-boxes that active mode or non-extended passive mode can require (such as rewriting the IP address in the PASV command response to implement NAT). And then, you have to wonder, if you have to substantially change the software in use at the client and server (and possibly even write custom code, per your Apache+$LANG suggestion), are those significant changes really worth it just to save on one extra port open on the firewall?
> And then, you have to wonder, if you have to substantially change the software in use at the client and server (and possibly even write custom code, per your Apache+$LANG suggestion), are those significant changes really worth it just to save on one extra port open on the firewall?
Oh, sorry, I didn't make it clear before. I'm fully admitting that in the case of record-oriented file requirements, especially on the client side, FTPS probably doesn't have a better solution available.
I was just noting that a vastly simplified firewall configuration that both doesn't require multiple ports opened (or to somehow read the PORT/PASV or EPRT/EPSV commands and open ports dynamically) is simpler, which is an advantage. It doesn't necessarily make up to the record-oriented storage problems that would have to be dealt with in those cases though.
> The most reasonable "update" to FTP would be to formally replace it with HTTP, because FTP is an awful protocol --- maybe the worst one in common use --- that deserved to die off decades ago.
The biggest problem with FTP[S] is that it interacts poorly with NAT. So maybe the problem is IPv4 and the solution is IPv6.
> FTP also interacts badly with firewalls and network administrators (both the competent and not competent versions of them).
Passive mode FTP works fine with firewalls that allow outgoing connections. Even active mode can work using something like Port Control Protocol.
It's only a problem when the network is tightly locked down. And when everyone's answer to that is to use TLS/443 for everything, how is that better than just leaving the unprivileged outgoing ports open? You end up allowing everything either way, but with the everything-over-port-443 outcome you can't even selectively block things anymore.
> It also interacts way too well with the NSA and ISP eavesdropping.
You could say the same about HTTP, which is why they both support TLS.
I agree with your sentiment, but have found that enterprise partners are flabbergasted when I recommend using HTTPS for file exchanges, because they believe SFTP is the only way to accomplish this. When we control both sides of the communication (via an on-premise virtual appliance) we always use HTTPS, and only support SFTP where a customer is wanting to push data to us and this is the only way they can do it (because there's usually an ops-adjacent team that manages all file exchanges and they use software that only does SFTP--as far as they know).
In your (Matasano/Latacora) experience, does SFTP register as a security risk / exposure, or does it check the box as long as they aren't using FTP/FTPS?
How about when SFTP is treated as an alternative exposed protocol for your existing daemon to speak, with your program embedding an "SFTP-app framework" and giving it a delegate router/handler module—much like you'd embed a web-app framework and give it a delegate router/handler module?
I'm mostly thinking of Erlang's `ssh_sftpd` application here, where your application can "drive" it by exposing a "file server" for it to talk to as if it was the filesystem.
That seems like the worst of both worlds: a custom application and an SFTP server exposed to the world. We would not be thrilled about this. We'd secure it if it had to exist and be secured (sometimes, financial providers use SFTP to do batch job submissions), but would never recommend it.
Picture a "web-app" like an APT package repo host. (I know the reference impls of ALT hosts are essentially static-site-generators, but ignore that, and imagine an impl that's a web-app with dynamic business logic to expose the relevant routes.)
Such a web-app can be served over plain HTTP. Not even an MITM can exploit an APT client, since it looks for specific files on the server and then checks those files' signatures against a local GPG database.
What would be wrong with having this web-app also be an "SFTP-app"? Or an "FTP-app"? Or even a "telnet-app"?
In all these cases, it'd just be a virtual filesystem being exposed over a wire protocol where the security of the wire protocol is immaterial.
That's more in line with what I was picturing, here. You know how (public) S3 files can be retrieved over HTTP, but also over BitTorrent? Picture adding SFTP to that, as another option. That kind of API.
---
For the situations where the service would benefit from security, though, you can still just obviate the need for relying on SSH's security specifically.
For example, you can expose your SSH server only over a VPN. (I know a lot of people do this, though I personally find it to be more of a hassle than just correctly configuring SSH.)
Or you can expose your SSH server behind an stunnel instance, thereby reducing your configuration work down to just "setting up a TLS-terminating load-balancer", that can then be shared between HTTPS and SSH traffic.
---
Or, if it's just that you don't trust the Erlang SSH impl, you could just hack up OpenSSH to act as an "SSH terminator", forwarding the SSH-channel flows out of each SSH connection, as regular (auth-headered) TCP connections to your backend. Sort of like an fCGI server for SSH. I'm pretty sure the only reason nobody has done this yet is that no language (other than Erlang) has batteries-included support for speaking the server-side of SFTP.
We have customers that batch upload data to our cloud service. We originally only offered HTTPS uploads but found a significant number of our customers really preferred SFTP workflows. My best guess as to why is that WinSCP provides a comfortable UI paradigm and there isn't an equivalent (well known) windows GUI for doing HTTPS uploads.
Our SFTP server is written in go and exposes a VFS that has nothing to do with anything on the local file system. It doesn't even allow you to download the files you just uploaded in the same session.
Of course things like the recent libssh CVEs should make anyone exposing non-standard ssh services to the internet a bit nervous.
Modern TLS is less complicated than SSH, but even if it weren't, running two competing cryptographic transports is more complicated than just running one.
SFTP also provides what is in effect a shell (or, charitably, a remote filesystem), baked into the protocol. For example, SFTP has SETSTAT, which lets you set file attributes. None of what SFTP allows you to do is particularly difficult in HTTP (some of them are made harder by browsers, but that argument doesn't cary any weight since SFTP already requires you to use something other than a browser), but at the same time, none of them need to be baked into a file transfer protocol, either.
The conversation about FTP always revolves around uploads because that's the thing browsers didn't actually do in the 1990s. (20 years later, browsers can pretty much do everything FTP does.)
