It might be a regional dialect. Growing up in the west coast, I have never heard "her-bal tea", but rather always heard "er-bull tea". Hearing the "h" sound in "herb" (as I do whenever I hear British (and Australian?) speakers speak) was always very jarring, until I realized that it was just that they pronounce that class of words differently than we do.

It's possible that in the eastern or southern US, they say the H in "herb".