You can actually see this behavior in the car when it runs into a difficult situation. It tells you it's asking for assistance or something similar, and pauses for a few seconds.

I also made the mistake of assuming the remote operator drives the car but if you watch Waymo's technical videos, it's clear that the AI is in control of the car all all times and the remote operator is just doing near real time labelling of what the car is seeing.