That's what silent mode and voicemail is for. For the caller, there is plausible deniability that they were interrupting anything, so it's up to the recipient to manage phone availability.