They can tell by which way you enter the car. This is problematic since women commonly put their purse in the passenger seat sometimes via the passenger door.
You can easily be off by meters or even 10s of meters in terms of the position of the phone.
You have no way to know the position of the car in relation.
You would need car sensors and per car interior data and even then it would likely confuse a driver holding a phone in their right hand and a passenger holding it in their left.
You talk about the phone keeping track of where the phone not only is but was historically in order to guess if you are driving and texting and I hear you want to design a broken system that is not even deterministically broken.
It also fails to account for people passing a phone around to someone else in the car.
I wouldn't be able to give my phone to a passenger because it would already have decided it is a a driver's phone.
Your users will only be able to use your phone if they can reverse engineer what their phone thinks they are doing.
Anyone who does not charge their phone will see car travel destroy their battery life.
One can trivially avoid such a feature by disabling GPS but most people won't have to worry about it because they just won't pay hundreds of dollars more for a phone that randomly doesn't work even in the passenger seat because it's incorrectly second guessing you.
