I'm also an iOS developer although I've never needed to use the Address Book API's. I always presumed that some sort of permission was required I'm really surprised it isn't. I actually think this is a bigger privacy concern than location access as not only are you giving away access to your contact details but everyone who has trusted you with theirs.