The worst offenders I have known about vanity sizing have generally been fairly small. My grandmother, for example, was without doubt a size 6 but would never buy anything that wasn't listed as a size 4. I have had two girlfriends who were similarly annoying--the clothing I bought them was clearly dead-on the correct size when they tried it on, but they wouldn't wear it because it was a size "too large" (they didn't last long as girlfriends).
By contrast, the smallest and largest women I have known simply wanted something that fit correctly regardless of the "labeled size".
It's a shame there can't be standard sizing. All this confusion just because we don't want to offend the overweight and obese.