If you're a C purist (or just a purist in general) then of course you're right, but I personally think that C-with-classes is clearer, easier for a layman to understand (many people who call themselves C programmers still get nervous when function pointers come out to play), and is usually worth the trade-offs. You can write a lot of very decent code this way and have a little bit of "the best of both worlds", and avoid getting into a lot of the admittedly very dark corners and nasty surprises that C++ has to offer...
That's probably a really big one in terms of reliability.
Also, the weird C++ behavior of calling copy constructors in the most unexpected places can have a very detrimental effect on performance.
Personally, I think the nature of C++ class definition leads people down the primrose path of trying to provide all potentially useful methods, of which only a tiny fraction get used. Tons of dead code appears in most C++ projects.
Faster compilation, more flexibility, less dependencies in code base (you don't have to define struct fields in headers), no problems with incompatible ABI, interface that can be directly called from almost any other programing language...
It seems to me that it DOES pretty trivially give you polymorphism; just change the function pointers in a struct and it behaves differently, while retaining the same type! Inheritance can be implemented, too, by copying function pointers and other struct contents around.
If you play with languages that use prototype-based OO like Lua and JavaScript, you can see pretty clearly how to implement such things. Implementing your own OO system in Lua is an experience I'd recommend for anyone who uses OO; it's easy, and you'll almost certainly learn something you can apply elsewhere.