You're right. The more fine-tuned it becomes, the more you have to consider low-level protocols. This paper https://nvlpubs.nist.gov/nistpubs/jres/121/jres.121.023.pdf suggests with current systems, time uncertainty can be decreased to about 1-10 us from NIST.