I've had a very similar experience. If i drink coffee daily, it bumps my anxiety significantly after a few days or a week. If i rarely have it, it's usually fine. But if I let myself have it, it's too easy to trend towards daily again.

I drank coffee for years and don't think it had an impact, but at some point, I noticed it and the impact was very real when not drinking it.