I mean running bind[1] locally configured to act as a DNS cache.
The operating system does some DNS caching as well, but it's not really tuned for crawling, and as a result it's very easy to end up spamming innocent DNS servers with an insane amount of lookup requests.
ok but in my understanding isnt DNS also cached on the nearest wifi/ISP routers? the whole DNS system is just layer after layer of caches right? i.e. does caching on local machine actually matter? (real question, i dont know)
Yeah sure, but most of those caching layers (including possibly on the ISP level) aren't really configured for the DNS flood a crawling operation may result in.
If you're going to do way more DNS lookups than are expected from your connection, it's a good custom to provide your own caching layer that's scaled accordingly.
Not doing so probably won't break anything, but it risks degrading the DNS performance of other people using the same resolvers.
