It is a two sided issue. As I am browsing the web I have referers disabled in the browser to protect my privacy. But at the same time I heavily on seeing where people came from to find my hobbyist website. It motivates me and I would not have discovered communities of like minded people.
It honestly wouldn't. I browse with some antivirus stripping out my referers and modifying other headers to be less identifying, and it's vanishingly rare for me to get any sort of problem whatsoever.
The flipside of this is that knowing where people have come from is very valuable even in non-invasive ways to business, and in the end that's what the internet is more or less about nowadays. The average web user probably only uses a few things which aren't solely controlled by one vendor - email being a notable one, and probably why Facebook messages will never win in the end if history teaches us anything :-)
