Hacker News new | past | comments | ask | show | jobs | submit login

I run a search engine crawler from my residential network. I get this too sometimes, but a lot of the time the IP shit-listing is temporary. It also seems to happen more often if you don't use a high enough crawl delay, ignore robots.txt, do deep crawls ignoring HTTP 429 errors and so on. You know, overall being a bad bot.

Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: