Cloudflare protected people from DDOS. They stopped abusive individuals from removing websites and their content from the Internet. Now Cloudflare is inventing new ways to prevent us from accessing information. They've become the people they swore they would fight. You either die young or live long enough to see yourself become the villain. The side that is good is the side that fights for knowledge and to make it plentiful and available to everyone, including robots. That's what's going to make society flourish. Not this scheming and rent-seeking. Building an empire that panders to resentfulness is like building on sand.

AI scrapers are, from the perspective of the website operator, indistinguishable from DDOS. I don't owe anyone any kind of special exception in my firewall.

You'd have to have the slowest site on Earth to not be able to serve legitimate crawlers. Have you ever truly been DDOS'd? I have. I actually had to start self-hosting my website because back when I used Cloudflare, the people who'd DDOS my site would just take down Cloudflare's servers. They're not even a very good protection racket. They're just in it for the money and power.

I have the opposite experience. I was not able to reliably keep my website online until I bit the bullet and moved over to Cloudflare (pre-AI).

> They're just in it for the money and power.

I would wager it's impossible to buy a product from a company that is not in it for the money and/or power. Especially in comparison to Microsoft, Google, Meta, etc.? I'm trying really hard to empathize with your point of view but I can't relate at all.

The point of a company is to provide a valuable necessary service to society. Money and power is simply a consequence of being more qualified to serve society in that niche better than anyone else. Cloudflare isn't qualified enough yet to be the people they're angling to be. They need to learn to be better people and how to do a much better job. Turning to villainy won't help them hit the mark after failing to meet expectations.

> Money and power is simply a consequence of being more qualified to serve society in that niche better than anyone else.

This may be one of the funniest sentences I've ever encountered on HN.

Bearing in mind, this was a decade ago, and the backing tech changed since then... but at the time, the site was mostly classified car ads. Each page delivery tended to have several dynamic SQL queries to deliver the page itself, but also related content, most popular content, etc.

There was no caching and really normalized data structures on the backend when I started. During my time there, crawlers/scrapers quickly became more than half the requests to the site. Going from about 1M page views per day to 30M was crushing the database servers... I took the time to denormalize and store most of the adverts, and some of the other data into MongoDB (later Elastic) in order to remove the query overhead in search results... It took a while to displace the rest as it meant changes to the onboarding/signup funnel to match. I also did a lot of query optimizations, added caching to various data requests and improved a lot of other things.

That said, at the time, the requests were knocking over a $10k/month database server. Not every site is setup as static content... even if a lot of that content can and should be cached. All to service a bunch of crawlers that delivered no eyes and no value to the site.

They were DDOS protection first, then expanded into edge caches and reverse proxies. Back then, they did not offer paid services to DDOSers to bypass their protection, or if they did, they were at least discrete about it.

Actually they were a honeypot first, or so I'm told. https://www.youtube.com/watch?v=RxhZ2vOjF5s

Ironically the AI crawlers I do want to block - the million-IP-strong residential botnets that fake their user agents - Cloudflare doesn't detect at all.

As an operator, I have questions about this; I also have very good metrics. I see a lot of what looks like what has traditionally been SYN reflection attacks. I have solid metrics and TTPs, which I'm willing to share TLP:RED and possibly discuss TLP:YELLOW.

I'd like to see some metrics which compare proven bot activity vs SYN reflection against the same infrastructure.

"the million-IP-strong residential botnets"

do you understand how much money to get this???? or are you implying cloudflare is failed to do its job since its not reaching 100% foolproof ????

this is crazy and you are free to use alternative that better than that

wait a minute there is none!!!, turns out a magic silver bullet software that offer 100% protection is NOT EXIST

You’re saying that Cloudflare’s capabilities are wildly overstated? Apostasy. In this forum, nothing ill must be said about their lame technology. You are only allowed to make vague complaints about their role in society.