My only thought is even with password auth. turned off hostile persons can still connect to the ssh daemon and might discover and unknown vulnerability or some type of dos. If you know someone is trouble why let them in even if you are wearing armor?