I had my GCP quota algorithmically set to 0 after spending 6 months working with them to launch a startup.

I went through a ton of hoops to get approval for our quota. We sent them system diagrams, code samples, financial reports, growth predictions, etc. It was months of back and forth. I'll also add that it was very annoying because they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks. In any case, after 6 months, they eventually approved us for our quota, we launched, and they shut us down to 0 quota across all services the instant our production app got traffic.

We contacted them again asking for help. We never got any human response. We got a boiler plate template a few times, but that was it.

I will never ever ever again use a cloud service where I can't guarantee that I can get good customer service. Unfortunately for a small business that means no big clouds like AWS, GCP, etc.

Yes, I am bitter.

Has AWS support gone downhill in the last two years? I've worked with them in the past - as both an individual and a couple startups - I always reached a human. Issues weren't always resolved as quickly as I'd like but response times were short.

Working in small and medium businesses I've observed the same thing, and I've been quite satisfied with it. So I don't think it's really gone downhill, so GP's comment doesn't really resonate for me, but that isn't to negate their experience. Otoh I keep hearing horror stories about GCP and now I'm reluctant to try it.

Shitting on GCP is just popular on HN and always gets upvoted. AWS and Azure have royally fucked thousands of customers if you care to search for those writeups. My wild ass guess, considering posts like these have zero background details, is that they were careless with service account keys and their account got suspended for mining crypto or something. They also probably weren’t actually paying for support of any kind and that’s why no one is responding to them.

Nope. We had been testing in our development and staging environments for months. We were deploying to production the exact same stack and we got our quota revoked within about an hour. We must have tripped some random thing. We have absolutely no idea what I could have been though.

I have the same question. I always got human response after 24-48hours or after one round of messages (with an automated human or machine, not sure). But so far, across 3 accounts and a dozens of correspondence, I always got a human.

Last time I checked you had to pay something like $400/mo extra for legit, timely human support from AWS.

It has!

In the 2010s I always got an AWS support team to help.

Now I get handed off to an external partner of AWS certified contractors.

They are often terrible. They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk". Basically trying to find an issue in my pipeline. Which they never do because it's the same TF scripts used for years.

Only once we waste time going through the motions do I get passed up to someone who can actually correct the backend issue in the AWS stack itself.

Tbf though I rarely ever have to contact AWS support at this point. The few times I have in the last 2-3 was due to issues after they rolled out an update or with a newer service we wanted to use.

Never have issues with stable services like S3, ECS, EKS, or RDS.

> They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk"

To be fair I would bet money that the overwhelmingly vast majority of support tickets are exactly those kind of issues, and ones that refer to actual bugs on their end are, comparatively, extremely rare, and should have to be escalated through normal procedures to weed out common problems.

if you want best support (while staying with big cloud) then Microsoft is the best .

Azure has its flaws but Microsoft puts a lot of people and effort behind it . We are not that large but there are so many instances where Microsoft reps will come in call with our customers or their people working with common customers will help out etc.

AWS has a done a decent job of taking enterprise business seriously last 10 years. you can get human support but generally they will charge you , I.e if better support you want you have to pay for premium support plans .

They are constrained unlike MS they don’t have non-cloud large enterprise business relationships for decades M365 or AD etc that helps with building the enterprise DNA.

In all three clouds it works best if you don’t buy directly, buy through a partner reseller , who both have the relationships to the CSP and have the people to work with you .

Sorry, no.

MS is the same network were even their lead engineers answer "well, uhh create a new account and hope you're not banned", when it comes to fixing a illegitimate ban issue.

None of the biggies are good. None of them.

You're better off building your own data enter. Can't believe I'm saying that, but I am. And it doesnt have to be acres and MW and water cooled. It can be a 42U rack.

Hell, I'm a homeowner and have 27U rack with 10U full, battery backup, solar, fiber and a backup internet connection, and stuff.

A small business could easy do this and own the hardware and software to their enterprise. In fact, they probably should. Helps prevent rug pulls!

Now I am curious what is the realistic price a business would expect to put down for a full rack. Say UPS, switch, 4-8U storage, and the rest CPU compute. Without entertaining GPUs, I bet you can get very respectably speced 1-2U servers for $5k a pop. So few hundred thousand probably gets you just an unbelievable amount of horsepower.

Having run a small business on one of the big clouds for almost 10 years now building your own data center is insane advice.

>easy

Hell no

Quota for what? In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.

The quota we needed increased far beyond the usual was the YouTube API. The startup was a media editing and publishing tool, with a feature to upload videos to YouTube on your behalf. Uploading a video requires a ton of quota, which they gave us.

Regardless, dropping all quotas to 0 effectively killed our GCP account.

Interesting. I guess we’ve learned an important lesson in not building businesses around APIs that don’t have an SLA…

How many services have meaningful SLAs for extreme downtime?

Github and (parts of) AWS will give you a small discount at 0.1% downtime, a bigger discount at 1% downtime, and AWS will refund the whole month for 5% downtime. But beyond that they don't care. If a particular customer gets no service at all then their entire $0 gets refunded and that's it.

That was just a feature of the product to be helpful. Not a core function at all.

> Quota for what?

Sure, I'm interested too.

> In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.

Well 0 is not sensible, and who cares if it's weird if they got detailed approval and they're paying for it.

"... and they're paying for it..." - that might be the exact issue. Google has no way to ensure that these small shops and startups will pay their bill, so quotas are used to prevent the company from running up a large bill they won't be able to pay.

I see a bunch of threads on reddit about startups accidentally going way over budget and then asking for credits back.

This doesn't at all mean the startups have bad intent, but things happen and Google doesn't want to deal with a huge collection issue.

If someone rolled up to your gas station and wanted to pump 10,000 gallons of gas but only pay you next month - would you allow it?

Well that is kind of a problem of their own making. The clouds refuse to entertain the prospect of pre-paying for services/having some sort of hard spending limits because they know that over-allocation is probably driving a decent amount of revenue.

I dont really understand ops problem as I've been able to set monthly limits on expenditure. Seems trivial to setup.

Are you sure you're not thinking of billing alerts? AFAIK there is still no way to set spending limits at GCP.

That’s not how quotas work in GCP. Google sets quotas for certain APIs for interacting with GCP itself, like how many VMs you can create per second. They’re not billable. Sometimes these quotas can be be increased if you need them to be. But the way op described it makes no sense.

Sure, but the comment is so vague I’m skeptical the OP knew what they were doing in the first place, or it happened exactly as they wrote. Maybe a service quota was reset to the default? But just set to zero? Doesn’t pass the sniff test.

What did your account manager say about this. Getting this interaction right is the core of their job, enabling your business on the platform so you spend more money. With this bad an interaction I'd have asked for a new account manager.

A colleague had a similar quota issue. 4 times quota restoration request was rejected. Upon the final request he put “women owned startup helping underprivileged kids” and it was approved.

It can’t hurt.

> they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks

It boggles my mind anyone would base their business on their good will. By now it should be obvious that companies with a huge number of customers don't care about individual cases that much for obvious reasons. That's why they cut on customer support. You get much better support with smaller companies where you (as an individual or business) are much more important to them.

> I am a CS researcher at UC Berkeley. This has seriously impacted my work.

I would try to get help from your department. Somewhere within CS and CS-adjacent departments at Berkeley there’s likely to be someone with an official or unofficial connection to Google that can get you in touch with a human to at least clarify the situation.

So much information is missing from this.

What Google account? Is it personal Gmail? Or your academic account? Are you using this for personal reasons or professional or commercial reasons? What kind of payment method is attached? What was your level of usage? Any idea why you were suspended initially?

Because it could be that Google is reviewing your appeal and simply shadow-denying it, and you haven't provided the right information to make it look legit. E.g. if they think you're a spammer or mining crypto or they think you're creating additional free accounts to use free credits, they're obviously not going to tell you what makes them think that.

But if this is for university-related work, and your university purchases IT+cloud services from Google (as they probably do), talk to your IT department so they can get you in touch with their institution-level support. Obviously, for the attached Google sales rep, the last thing they want is a CS researcher losing access to GCP.

Also, why wait 2 years to take it public?

> I am a CS researcher at UC Berkeley. This has seriously impacted my work.

Can I suggest a topic for your next research? "Cloud exascalers and their negative impact on the society"

Or “Why I should have been paying for support if my entire career depended on it.”

There are so many things with this statement I don't even know where to start. I hope you're being sarcastic.

Not sure why this is getting downvoted. This may be snark, but this is 100% needed in the world we live in today. It is a fact of today's world that individuals have no leverage over these companies. I can understand why big companies, who have leverage, buy their services. But I don't understand why individuals, who have no leverage, buy their services and build their profession and livelihood around them. Any day, they can cut you off from their services. You are being irresponsible to yourself if you put all your eggs in these big tech baskets.

We seriously do need this kind of research and compelling articles that argue why relying on these big tech cloud services is harmful for individuals.

Unless you know someone at Google with a little swagger to get things done you’re shit out of luck.

Our GCP account manager was always pretty good at solving these sorts of problems for us at my last company.

The role of GCP is to help enterprises negotiate better deals out of Azure/AWS. Why would anyone actually use it is beyond me...

Did you tweet them - https://x.com/googlecloud?

If it's so important, maybe talk with a lawyer.

It's gone. No human will ever respond to you. That's how these companies operate. From here, you realistically have two options.

1. Forget the account and move on. You could create a new one, but nobody can tell how long it would take before that gets suspended as well.

2. If the suspension has a tangible negative impact on your profession, hire a lawyer and get proper legal advice.

Most important of all, let this be a lesson for you and your colleagues. It is a terrible idea to let any critical part of your life depend on unregulated industries that can wipe out someone's livelihood at the whim of machine learning systems. Learn this lesson and pass it on to everyone you know.

As an individual, you are nobody to Google and you have no leverage. It is reckless to build your livelihood or profession around their platforms. If you were a company, your team could speak to an account manager and negotiate. As an individual, your only real leverage is legal action.

Stories like this appear every month. I don't know how many more it will take before it becomes best practice not to depend on these utterly abominable rackets for anything critical.

> let this be a lesson for you and your colleagues.

Nah, big tech infiltrates everything, it’s 100% their fault. Why did everyone switch to webmail? Why did we gravitate to web apps? Big tech persuaded us all to do it.

With big promises comes great responsibility, and the stuff in the fine print doesn’t count. It’s not ethical to invite dependency and randomly kneecap people; it shouldn’t be legal either.

100% agree with you. The big techs are definitely 100% at fault. But you know, fool me once, shame on me. Fool me twice...

I mean, we get these stories every month. Yes, 100% it is not ethical to randomly kneecap people. But let's be honest. Nobody is working on making these big tech companies accountable for the potentially devastating, algorithm-driven decisions they take. How many more times do they have to fool us before we all realize that it's time to move away from them?

All I ask from you, myself and all the tech folks here is to learn from these lessons and pass them on to everyone around you. With how things are today, it is reckless to depend on these big tech cloud services for your livelihood and profession. If you're working for a company where the company has leverage, all good. But as an individual, you should stay away from these big tech companies, because they can screw up your life any day, without warning and without recourse.

With email in particular, it's not like chocolate. If you self-host email, or use any other cloud host besides Microsoft or Google, then Microsoft and Google will randomly fail to deliver emails you send to their users, even though you have SPF, DKIM, DMARC, etc. set up exactly right.

Seriously #2 is your only recourse. Download the terms of service / your service contract , highlight their violations and send them a certified letter about breach of contract and that you intend legal recourse.

Cloud lock-in is real and requires a lot of forethought to avoid or at least mitigate. on the LLM side I have pushed my last two companies to always have at least 3 vendors for hosting and a system to fail over instantly or even load balance based on different criteria. It has paid massive dividends. I wish that philosophy was easier to implement at the cloud level for all services.

How come your business or research are so tied to GCP? What about other providers?

It's not asinine. This suspension happened two years ago. I don't see any sudden panicking.

UC Berkeley gets much of its IT infrastructure from Google in a big and expensive contract. Perhaps some of the campus staff could try to negotiate on your behalf?

anything that has to do with google avoid at major cost

or have an alternative ready

for serious work -- don't use google & don't use google devices either

Plenty of GCP, AWS and Azure experiences in this thread, but any Cloudflare experiences?

I had a bad Cloudflare experience. So, my card on file got no balance one day (my bad, I forgot to update to a new card), and they just turned off the services.

They somehow managed to charge partial amount (like 80% of the bill), but decided to turn off everything anyway, even the services that could be covered by those 80%. They turned off what they offer for free, and we were unable to change the setting, like instead of their CDN point traffic to an S3 bucket, etc.

When they do that they basically freeze your account. I mean you cannot provide a new card to pay the outstanding bill, or do anything at all actually. You're not welcomed here anymore. Locked out. That's is a terrible way to react to a payment failure after being a paying customer for a few years.

It was hard to reach the support, and it took multiple days until I found someone on Reddit who looked at our ticket and it eventually helped.

PS I had much worse experience with GCP after being a loyal customer of them for like 15 years, so Clouflare is good.

I am going through a very similar issue with Cloudflare right now, and billing support is almost of no help.

Yeah, we were looking for image CDN services (with resizing etc). Asked CloudFlare and they said $200 a month, everyone else was saying $3-5k per month.

Had a sales call with CloudFlare, they said yes they do flat rate billing and it's only $200 a month for all we can eat image hosting.

We of course called bullshit and third time around (talking to human sales reps) we said, just to get it in writing, we can do X bandwidth/Y images for $200 a month?

...oh errr, no, that would be more like $7k.

Thankfully we smelled bullshit and didn't take sales word for it. We'd have built an integration and started paying only to be bitten a month or two later when they readjusted our pricing. They basically refuse to talk about real pricing until you're already paying $200/m and locked in.

We ended up hosting our own on GKE for $500-$1k/m.

GCP fired my neighbor just before they were about to receive a promotion by attrition fill, but instead was replaced by offshore, lower tier support.

The Lord giveth, and the Lord taketh away. Blessed be the name of the Lord. - Job 1:21

This is such a repetitious issue that I wonder why there has been no class action suits so far?

I think documenting these cases somewhere, and targeting not just Alphabet but all the other "we're too big to support little people like you" companies would be a good idea. I don't think the pay out would be significant, but the punitive impact might change things.

OP is not clear , but it looks like GCP suspension not Google one (I.e email android etc)

All clouds reject a lot of businesses for their services for variety of reasons and there are alternatives in the market unlike say a Google account suspension .

I don’t think class action is feasible for cloud computing suspension (unless of course they are discriminating against a protected class etc)

i was thinking more in terms of tort law or contract law. They probably have a disclaimer and Tos that addresses all that, but given enough plaintiffs and their market dominance, it might amount to possible deliberate/calculated financial harm. It might be enough to not get thrown out of court at least. They can reject business for any reason, but once someone relies on their services for their business, there is always a certain expectation of continued service, and in the event of service termination, they may not need to explain themselves, but they must accommodate reasonable requests to transfer data, customers,etc.. elsewhere. Otherwise it sounds like tortiuous interference.

https://en.wikipedia.org/wiki/Tortious_interference

> Tortious interference, also known as intentional interference with contractual relations, in the common law of torts, occurs when one person intentionally damages someone else's contractual or business relationships with a third party, causing economic harm.

In this case, people who use GCP have customers and other contractual relationships. Google's termination of service interfered with that. Google also doing this as a matter of standard business practice indicates that they are aware that their action will interfere with people's contractual obligations (well common sense should tell them that anyways).

You can't force someone to sign a contract with you that says "if I interfere with your future contracts with arbitrary third parties on purpose, you can't sue me". The deliberate part is crucial from what I understand. If their decision making couldn't have accounted for the interference, and the interference wasn't calculated as an acceptable risk, there is no issue. But the plaintiffs can claim that repeated social media posts and acknowledgements of said interference by Google over the years means it's enough grounds for a suit. and a suit will mean discovery, google will have go hand over internal documents, depose employees,etc...

In the end, this might be more costly to companies like Google than just giving customers a grace period to move elsewhere before termination.

Obligatory: IANAL, I'm just a guy using big words I barely understand.

Of all the posts like this I’ve seen the customers are always 1) extremely scant on details about what they were using GCP for or why they were suspended, and more importantly 2) never actually paying for support.

Having worked with a fair few academics, I’m guessing they lost track of their service account keys and the account got suspended for crypto mining.

There have been plenty of posts where the reason was apparent. One i recall was caused by a guy having malware on his phone, and he happened to use a work email on his phone, so the entire GWS organization was banned, shutting down the company's operations.

I have yet to see someone say why they were suspended.

I’ve always wondered why, this makes sense.

> but the punitive impact might change things

Call me cynical but I have little to no hope that even class actions would solve anything. These companies have become so big that they can take one class action after another for years to come without making a dent in their financials and without bringing any change to their operating procedures.

I'm just expecting them to change their calculus. Right now it costs them nothing to randomly shut down accounts. If it had some cost, perhaps some minor notice, accommodation,etc.. however automated might be worth just the man hours spent on lawsuits.

One of the many reasons I continue to degoogle and remove that garbage from my life wherever I can. So many cases like this.

Xbox & Discord are the only 2 services I’ve seen handle bans with adequate transparency (yes there is still room to improve). Both offer a ban status tab ranging from hand-slap to giga-banned , allowing you to have some level of warning before being booted.

Given how dependent we all are on these services: we run our businesses and our lives, it’s despicable that more due process and transparency is not offered for shadow and proper bans like this.