Well clearly you wouldn't keep your keys in s3, if you were to keep them on an ec2 instance, for example, as long as you had proper SSH access configured even with AWS account access the servers would still be secure.

That then makes your local SSH key the last link in the chain (and OpenSSL of course, and that you trust AWS is actually encrypting the data).