Time is not a dimension of the same kind as spatial dimensions. It has a different metric and you can’t move freely back and forth on it. When you rotate on the XT plane, it doesn’t mean the same thing as rotating on the XY plane. It is not a good candidate for the sort of fourth dimension we’re interested in.
My understanding is that time can be a 4th dimension, but n-dimensional spaces themselves are simply a very basic mathematical structure, where a point can be described by n numbers (you can actually be abstract even in that, no need to stick to rational numbers, I believe).
As long as you can map time to a number line, it's a valid representation. We just happen to have hardware acceleration for 3-dimensions, and the 4th is just completely unintuitive to us.
If we're only talking about simple vector spaces, your understanding is accurate, but when we're talking about visualizing shapes in 4 dimensions, we typically want something more. We are doing geometry then, and so we want a metric space that defines a concept of distance (which vector space don't have).
When it comes to geometry and not just vector spaces, time dimensions have a different definition of distance than do space dimensions. There's a minus in the formula where you would usually have a plus. And this means that shapes in this space behave very differently than what we're after when imagining a hypercube or hypersphere, for example.
We want to think of a 4 dimensional space where all the dimensions are indistinguishable, but the minus sign in the metric distinctly identifies the time dimension. For this reason, physicists typically call this kind of space a 3+1 dimensional space rather than a 4 dimensional one.
