Assuming openssh-server, look for "internal-sftp" in sshd_config(5). And of course, also google a bit around that.

It's worth pointing out that this will preclude use of rsync and similar tools that require running a server on the remote machine. I'm not aware of robust alternatives so if anyone else knows one please tell me! (and I consider rsync-restricted shells and similar approaches a hack)