I think these permissions should be handled in a way where the app simply cannot know whether you gave limited or full access. The app should see a pool of items (contacts / photos / health data / whatever) without the information whether that's full access or not.
If I understand correctly, apps are only given that info to be able to add a custom `Adjust Selection…` button in a convenient place in the app. But this could be handled on a system level instead, where a small unobtrusive pop-up appears, on top of the app without its knowledge, where the user can adjust permissions (kind of like the `Pasted from…` toast when using the clipboard).
It's a bit uglier, but much safer and avoids digging into Settings like you mention. And crummy apps cannot bully you for giving partial access.