My understanding is that time can be a 4th dimension, but n-dimensional spaces themselves are simply a very basic mathematical structure, where a point can be described by n numbers (you can actually be abstract even in that, no need to stick to rational numbers, I believe).
As long as you can map time to a number line, it's a valid representation. We just happen to have hardware acceleration for 3-dimensions, and the 4th is just completely unintuitive to us.
If we're only talking about simple vector spaces, your understanding is accurate, but when we're talking about visualizing shapes in 4 dimensions, we typically want something more. We are doing geometry then, and so we want a metric space that defines a concept of distance (which vector space don't have).
When it comes to geometry and not just vector spaces, time dimensions have a different definition of distance than do space dimensions. There's a minus in the formula where you would usually have a plus. And this means that shapes in this space behave very differently than what we're after when imagining a hypercube or hypersphere, for example.
We want to think of a 4 dimensional space where all the dimensions are indistinguishable, but the minus sign in the metric distinctly identifies the time dimension. For this reason, physicists typically call this kind of space a 3+1 dimensional space rather than a 4 dimensional one.
As long as you can map time to a number line, it's a valid representation. We just happen to have hardware acceleration for 3-dimensions, and the 4th is just completely unintuitive to us.