This is actually a problem for my Volkswagen Tiguan, the car GPS data is wildly inaccurate probably ~30% of the time. The only fix is force it to use wired CarPlay or turn of CarPlay and let my map grab my phone's data before reconnecting.
It does! Unfortunately for me, I rented this one Ford Explorer where the car's GPS was off by a mile, so that was the opposite of helpful, and I didn't find a way to turn that off in the short time I had it.