> * By a huge margin the nosiest device is my Android Phone. >40% traffic is coming from just my phone (which is crazy!)

I work for a small ISP-for-schools. We had an issue which eventually turned out to be related to a specific version of Snapchat on Android, when its connection back to Snapchat was blocked, it'd try and send a mixture of GET/POST at a rate of 1000s a minute. When you've got thousands of devices doing that, it's like an internal DDoS.

We've had a few problems like this and it always appears to be Android apps...

How did you resolve that snapchat problem?

I wanted to say something positive too, after seeing how the thread went here. No issues from the non-technical users in the household, at all, really. The domains I've whitelisted are mainly one-offs - I haven't whitelisted GA as above and nothing has broken here.

I've ended up using uMatrix and uBlock together on my personal machine as they don't interfere with each other and uBlock has an extra list or two that blocks even more tracking and ads - including all youtube ads. Which is just so satisfying. But other devices (of all kinds) run ad and tracker free, with a faster browsing experience to boot.

So that's no ads on any devices on the network, all running on a pi that is also a media centre, NAS and IRC client. Takes minutes to set up and is regularly updated. I don't get what's not to like.

When something isn't working, how do you determine with a tiny bit of effort which one or more domains out of the 650,000+ domains to whitelist?

Here is what I do:

1. Switch off DNS from PiHole to ISP DNS (or Google/CF, whatever non blocking DNS)

2. Reload page in Chrome with uBlock Origin and selectively allow domains to load until the page loads.

3. Whitelist domain in PiHole

This works b/c both uBlock Origin and PiHole out of the box use basically the same blocklists (or you can force them to).

How often do you find yourself doing this? Probably hard to determine, how frequent when you first set it up, vs after a month of use? Thanks

Thanks. Can you expand on what you mean in step 2?

What is involved in selectively allowing domains to load?

Could you do it if the domain is a completely unrelated string to whatever site you are visiting? (Say for site example.com, it requires something from whwehkhsfasfs.com in order to load)... how does step 2 work exactly in this case? Are you being prompted for a small subset of domains that the page is trying to load, for example?

So, PiHole has a web interface to blacklist/whitelist items, but it's hard to use for debugging as what it "sees" are just a bunch of DNS requests come through (they aren't grouped by page/user - at least in the version I had going).

But in Chrome with uBlock Origin - it very clearly tells you what's happening and you can selective unblock domains until the page starts working and then turn around and add that domain to the whitelist.

I'm making this sound harder than it actually is, it's honestly just a couple mouse clicks and page refreshes in Chrome with uBlock origin going.

I've always found it trivial to know what to whitelist.

Pi-hole admin -> Query log -> Show all -> Search filter <domain>

Whitelist whatever necessary.

I use the web interface and filter it with the IP address of the device. It will usually show up in the top of the list and try whitelisting.

Example my Samsung TV took some effort : multiple domains were blocked and had to restart my TV everytime, thankfully PiHole has a neat responsive web interface. After allowing 2 domains it started to work or else it wouldnt go online.

Easiest way: check the log to see what was blocked. It's as simple as that.

Happens so infrequently though - the lists are made up of domains you don't typically want to whitelist. Only 133,608 on my lists though, and that's pretty up to date.

I use AdAway on my phone to blackhole ads and trackers. I've seen some weird behavior in certain apps I assume is due to it but I've never had any app not work.

Same thing on laptop. I use a combination of blackholed DNS, a firewall, and uBlock Origin in the browser. Some sites won't work due to poor error handling which is their loss. I've never had a native app not work.

> which is their loss

Exactly what my stance is towards sites that 'need' crap to work. My favorite are news sites/blogs whose primary content is text, but display a completely white blank page when the adblocker is up. (some even do this when you disable JS too. why JS is 'required' for displaying any text is beyond me)

> why JS is 'required' for displaying any text is beyond me

It is not. But this is likely a result of either:

1) simply not realizing that the newfangled 'framework' they are using requires JS to render text

2) intentionally requiring JS to render text, because then JS will be on so they can also deliver ads (i.e., a form of "anti-adblock").

3) not all readers just want plain text and there are JS based features that actually appeal to readers.

Things like backgrounded next page loading may not seem appealing to you, but stuff like that does fall into the category of "people actually like this" and not "implemented to intentionally force ads on people".

We use an express server to pre-render our text, which means you can always see the content, but when you strip out JS, you lose a lot of navigation perks.

I fully support you not wanting JS, but don't act like opting to use a ubiquitous tool is somehow either ignorant or malicious.

> We use an express server to pre-render our text

That's silly. Unless your users are on throttled dial-up inet, and you're trying to feed them several volumes worth of text (a MB or so?), pre-rendering text on the next page is not a good enough excuse. This must mean you are using some large framework to deliver text and present your website. HN seems to be able to deliver tons of text to users without javascript bullshit, why can't you?

Cost, I would imagine. It's there. It's working for most people. The "only people complaining" are those who want to take what you have to offer and stop you from getting advertising revenue for seeing it.

Your argument, whilst full of technical merit and other benefits which you have not mentioned, is perhaps not persuasive enough to those who control the purse strings.

I mean we pre-render our text before it hits the browser. So if you have JS disabled, you still get text.

Also, HN has a completely different userbase than most websites. The biggest complaint about Android in the early days was "it has UI/UX for engineers" (that I personally loved). 99% of people aren't looking for a website experience like HN. Most people WANT some sort of slight flashiness and style. Also, most people don't use noscript. It's a valid tool to use, even if it is abused by some. We still run tests to make sure our total delivered payload is small and monitor accessibility stuff. Just because some sites are built like a dumpster fire using modern frameworks doesn't mean all modern frameworks are bad.

There is a huge difference between "you have completely abused javascript and now I get a blank white page with JS off" and "you've used JS to make interacting with your page much nicer"

Thank you for not breaking the non-JS use case, and thanks for elaborating on your point!

> It is not.

That's why I put 'required' in quotes, indicating that they think it is (for the reasons you pointed out) but it's obviously not.

So this happened after a posted this comment (weird coincidence :/)

My pi hole graph started to look like - https://imgur.com/a/Jj8bmPW

NVIDIA geforce experience really wants to talk to the mothership :| Ended up uninstalling it

I wonder how tricky it'd be to serve a local copy of google analytics code that simply didn't report back to them. Or perhaps just redirect outgoing requests from GA to some internal resource that collects (for yourself instead of google) or drops the data and returns an expected response.

Then there would be no need to unblock their trackers to make websites function.

Due to SSL, it wouldn't be easy: you'd need to have SSL certs for some google.com subdomains, and they'd need to be trusted by every device. Blocking the traffic is much easier.

You're right that you'd need to trust the certificates on each machine, but really, you only need to trust a single CA on each device. With an SSL-replacing proxy you create and give control of a CA certificate and key to the proxy and set it between your computer and the network. It replaces the certificate of any (or some small subset if you want to specifically target) site with its own on the fly. Then it can inspect and manipulate the contents of that communication. If the machine trusts the CA, and the CA trusts the certificate (because it replaced and signed the cert itself.. so not so much trust, but.. you get the idea. You could probably limit it to certs signed by CA known by your browsers), the machines would trust the certificates.

Enforcing use of the proxy could even be automatic if your router supports it. eg, LEDE can redirect all traffic outbound to WAN on 443 through a specific endpoint (your MITM proxy or pi-hole or whatever).

It might be problematic for guests who have never seen the CA before, but that's what guest networks are for, I guess.

I've seen this technique used by some large multinational and security-sensitive companies to help monitor data egress from their networks. Probably via some overly-expensive software, but the software doesn't have to be expensive. They tend to have better automation systems than your typical home user, though.

However, for a small network, it's fairly straight-forward to get a CA certificate onto each device. If it's you or a few people on individual machines you or a few people you can add it manually in less than a minute on each. Or for the larger case there are automation tools.

One such open source project for an SSL replacing proxy:

https://mitmproxy.org/

> Write powerful addons and script mitmproxy with mitmdump. The scripting API offers full control over mitmproxy and makes it possible to automatically modify messages, redirect traffic, visualize messages, or implement custom commands

I'm sure there's a way to make this live side-by-side with pi-hole or something similar, but I unfortunately have other things on my plate.. Would make an interesting weekend project someday, though.

The only way to do it is to have admin/root on all the devices so you can install your own SSL root certs and "steal" google's domain internally with suitable certs.

It can be done but not in a BYOD situation.

Sounds a little like the “decentraleyes” extension...

https://git.synz.io/Synzvato/decentraleyes

I've been thinking about setting up twilio with PiHole so I can just copy-paste-sms a url to unblock for people on my network.

Any idea of the feasibility of this?

pihole has a CLI to block/unblock URLs, so it should be pretty straightforward

+1 The API support this too

Would you mind sharing your lists? I only have about 140k in mine but would like to increase my protection.

Firebog.net has curated lists

How did you detect which device was the nosiest? Do you have static ip addresses for each one?

It's my home network, there are devices you can count on your finger. Also DHCP lease times by default are in multiple hours on home routers so IPs don't often change. also the web Interface shows hostnames

Which blocklists are you using? Could you share?

I use lists curate by firebog https://firebog.net/

Ultimately, this is going to end badly for advertisers. All ISPs block port 25. In < 5 years, all ISPs will provide PiHole functionality on their networks. "Net Neutrality" is gone. This is going to be glorious. Thank goodness for the FCC making this protection available to the masses.

Or.. Advertisers will pay ISP's to inject ads.

Care to elaborate?

Web ads are the new spam. In response to email spam, all ISPs blocked port 25 by default. ISPs will ultimately do the same to web ads using traffic prioritization techniques like PiHole.

Users in general aren't sophisticated enough to do this themselves. ISPs will offer it for a price and users will pay more to have ad free internet while simultaneously benefiting ISPs with bandwidth reductions. Some ads will be allowed through of course, but with significant costs associated with the advertising, the volume will be reduced significantly, like the difference between spam email volume vs that of postal junk mail.

If big sites like Facebook decide to fight it in a cat and mouse game, ISPs can hit them with advertising fees directly or throttle their traffic in retaliation for cheating the system. If browsers like FireFox try to defeat it by doing DNS over HTTPS, then ISPs can funnel 1.1.1.1 directly into the trash. AT&T did this and said it was an accident. I'm sure it was more like a test. They are aware of what underhanded scheme FireFox is up to. Mozilla isn't fooling anyone with their "security" cover story.

I have no idea why you think ISPs would do something like that. American ISPs are anti-consumer to the hilt. They've been caught numerous times INJECTING ads into unencrypted http connections. Ads cause ISPs no issue whatsoever. Why would they block ads?

Also, Firefox has excellent extension support for browser-level adblocking. No ISPs I'm aware of do any kind of adblocking. If there is no ISP adblocking around, how on earth could DNS-over-HTTPS be a anti-adblocking move?!

I have no idea how you managed to convince yourself that ISPs are anti-ad pro-consumer crusaders while Mozilla are some kind of evil corporation trying to thwart their efforts. The reality is the exact opposite.