Ideally, I agree. Along with other freedom limitations depending on the crime (corruption should prevent you from actively managing your money/assets).

The issue is what to do for people without a house. But in a perfect society, you're right. We still are not there yet, so working on improving the current systems should be preferred.