Passwordless login for SSH not working

I have not been able to login to my system with passwordless SSH for some time now and finally figured out the problem.  I had to change the permissions of my home directory to disallow writing by everyone.  I knew that ~/.ssh and the files in it required specific permissions to be set, but I hadn't heard about the home directory having similar requirements.