Everyone: PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.

As the article points out: Every other cloud does this! They all have non-production subscription types with hard spending limits.

There's a difference between "unable" and "unwilling".

The people that refuse to understand the difference are the same people that don't understand that "unsupported" isn't synonymous with "cannot be made to function".

Don't be that person.

If you have a large account and you're in regular contact with an AWS sales representative, pressure them into making this happen. Even if you work for Megacorp with a $$$ budget, keep in mind that your future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.

Don't walk down the same path as IBM's mainframes, where no student anywhere ever has been able to learn on their own, making it a dead-end for corporations who pay billions for this platform. You, or your company will eventually pay this price if AWS keeps this IBM-like behaviour up.

Think of the big picture, not just your own immediate personal situation.

Apply this pressure yourself, because the students you want to hire next year can't.

I'm also going to point out, as a former AWS engineer, that "too hard" isn't in the AWS engineering vocabulary. If an AWS engineer were to dismiss something as "too hard", they'd be on the fast track to a performance-improvement-plan.

The problem isn't that it's too hard. It's that it isn't a business priority. As soon as it becomes prioritised, a legion of very smart AWS engineers will solve it.

As an antidote to the "nothing is too hard" flex...

Everything has trade-off's. When a responsible person says something is "too hard" it's because they've evaluated the consequences and have deemed them too costly in terms of time, resources, cost, maintenance, or strategy/lost-opportunity. Some things really are "too hard".

I agree it's not too hard technically but am surprised at your contradictory suggestion AWS would deploy "a legion of very smart engineers" to solve it if they decided to implement that ordinary feature. How smart are they?

I suspect the truth here is a bit more ugly. AWS doesn't want to do it because they see the pocket-change taken from accidental mishaps as a net positive. They don't care if it inflicts suffering on students and novices. Sure, they'll fix individual cases if they're shamed publicly or if the individual contacts them in exactly the right way, but at the end of the day, it's more money in their pocket. For every 200 dollar oopsie that gets charged back, how many go silently paid in humiliation and hardship?

I would speculate that Amazon doesn’t care about the occasional $200 mistake, and is perfectly happy to refund it should they notice.

What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.

I would guess most companies spending millions per month on AWS are just guessing what next month’s bill will be, and have been conditioned to not freak out if it’s a million over, as long as that doesn’t happen every month. That conditioning has to start early.

> What they don’t want is companies that are spending say $2-4 million per month to be able to put in a hard limit at $2.5M.

Why not? It would make companies happy.

The reality is that, as much as many would like this feature, I'm not sure that a lot of large customers are actually asking for it. Does anyone who works at a large company know if they've asked for this feature before?

I can imagine why startups might want it, but also, AWS is extremely forgiving with startups and throws tons of money at them, so idk.

The problem is, what to do when your account hits the limit? The subtle point about EC2 is you can’t actually shut down servers and have them come back up later unless you have a specific configuration - which not everybody has, so you can’t simply power off systems and not destroy something. Not insurmountable, but not trivial either. You also can’t say that there’s a limit except for an incompatible subset of products, nor can you ask users to be okay with randomly destroying things - the mere rumor of that destroys user trust in something. (Who’s still shy about using EBS despite the last major incident being back in… 2016? Also that outage was limited to one region.)

> nor can you ask users to be okay with randomly destroying things

what do you think will happen to your EC2 instances if your payment method on file with Amazon fails repeatedly and for more than 30 days? Do you think they will just keep those instances intact indefinitely without payment because they promised not to 'randomly destroy things'?

You are correct, which makes it even more pernicious that Amazon chooses to not fix this ... profits at any cost is not a long term benefit ... the longer AWS fails to fix this more folks will tend to lean toward the competition especially Azure ... I have been a unix/linux server side developer since the beginning and always use a linux laptop so no fan of Microsoft however their Azure platform is lightyears ahead of AWS ... its obvious the AWS console web front end needs a wholescale rewrite from the bottom up ... again as you say its not a priority probably because all big consumers never use the AWS console as they have written their own automation layer atop the AWS SDK

I routinely tell a coworker that <random task> isn't possible so that he does the work for me. Oh I know it's possible, I just don't want to do it. Thanks Corey! You're the best.

This works in help forums too. Post a question, there's a chance you get ignored. Post something wrong, and someone will flame you and answer your question.

Cunningham’s Law

(Had to resist the urge to post “Occam’s Razor” or some other incorrect thing to bait someone to correct me)

You should have tried that, would have gotten the point across even more effectively :)

As someone who tries to always get things right, it somehow seems wrong to deliberately post wrong answers.

I know it won't go over well at Stackoverflow, which is one of the very best places to get accurate and helpful information.

I had a smart coworker at one point, most of the time I asked for help with something, he would say that he did not know the answer.

Now I'm positive that he was doing the same thing as you describe.

The effect on me was a reduced interaction with that person, I'd much rather have an honest answer or a "I don't have time" answer.

This works until it doesn't, once Corey figures you out he might seek revenge...

score one for manipulation

This is in line with all the bad things I've heard about Amazon. Why do people work there?

Why is this particular thing bad?

"Too hard" is not an excuse. They have all the resources they need at their disposal to get the job done.

If there is no X for which you can say "X is too hard" without getting PIP'ed, then your work environment is dysfunctional.

Pretty sure is the ask was to solve the Halting Problem that you wouldn’t get in trouble.

What is opportunity cost?

Incalculable, thus irrelevant.

Scale that isn't available in most companies.

A culture they feel aligned with.

They are poor or high burn rate consumers.

They are curious.

But, it would seem, most of all they have different priorities that you.

I write that as someone who routinely dismisses their recruitment attempts.

The CV title of having worked at FAANG and the big tech compensation.

In the UK at least, Amazon pay relatively poorly compared to other the FAANG companies and Finance.

> "too hard" isn't in the AWS engineering vocabulary

Let these people fix climate change then.

Sarcasm aside, it must be boring working in a place where no problem is ever too hard.

a couple weeks ago Janet Yellen warned that net-zero carbon would cost about 2.5 trillion

that's cheaper than the pandemic. no reason it couldn't have been done already.

i no longer consider climate change a hard problem, the only hard problem is building political momentum without getting shot

I realize it can be fun to be snarky but consider the “business priority” portion you didn’t quote and the difference between “it’s too hard to do” and “doing it would cost X and impact performance of service Y by Z%”.

> Every other cloud does this! They all have non-production subscription types with hard spending limits.

They don't. I've looked. They all have some ultra complex scheme for monitoring billing and programmatically shutting down services, which is a favorite recommendation by apologists, but none have a nice, solid "shut down everything" plan that's usable for learning and testing.

AWS does have billing actions which are fairly new, but they're still a bit too difficult to deal with if you also follow the advice for setting up a root account for billing plus sub-accounts / organizations for use.

Azure has been ignoring the request for over 8 years [1]. There's a warning they're moving away from User Voice to a product-by-product solution for feedback, so that request might disappear altogether. I reached out to them and politely asked for an update and they said they'd look into it, but it's been over a month and I don't think there's going to be an update without some pressure. If anyone has a good social media following and would like to see hard caps on spending in Azure, tweet @AzureSupport. Maybe if enough people do they'll actually follow up.

> PLEASE stop making the argument that it's "too hard" or even "impossible" to implement spending limits.

I agree 100% with this because all I really want is something along the lines of the "personal learning" account that's described in the article. I want a dev / testing account where I can learn and test assumptions about billing without risking a massive overage that I wasn't expecting. They don't have to get into the complexities of dealing with production accounts.

AWS, Azure, and GCP have all had close to a decade to figure it out, so I think the only way it'll happen is if we start asking legislators to regulate those providers. It would be relatively simple in my opinion. Cloud infrastructure providers must allow users to indicate a maximum amount of spending per month and may not charge more than the indicated amount.

I think big tech is proving they can't self regulate, so it's time to quit letting them.

1. https://feedback.azure.com/forums/170030-signup-and-billing/...

Annoyingly, Azure does have this capability! If your company is a Microsoft gold partner (there might be other requirements, I can't remember) then your Azure developer licenses comes with £100 free credit per month. When this money runs out, every service associated with that subscription just shuts down. It doesn't overrun, heck I don't remember even getting an email about it, they just stop until it renews the next month.

They just refuse to implement this system for non dev licenses.

That's exactly the same way that Azure for Students works, which is why I use it instead of AWS Educate (well, that and because Azure doesn't limit what services you can use your $100 on, unlike AWS).

I even got a spending limit of 140€ / m? As dev in org

btw. azure has a subscription type where you can put a prepaid amount onto your account. they just do not support it by default. btw. bizspark customers get this of subscription, for 3 years with an free amount of up to 300 USD per month

IBM Cloud has Lite accounts, which are quite limited but always free.

"Every other cloud does this! They all have non-production subscription types with hard spending limits."

This is simply not true, to the best of my knowledge. GCP has budget limits, which can send alerts, but outside of manually catching a pubsub and shutting down billing on your own, there are no configurable hard billing limits.

I'm unsure what to make of the rest of your rant.

You can write a cloud function to automatically run when you go over the budget and shut down billing. They provide code for doing so in the docs:

https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

I'm not sure if that's what you meant by "manually catching a pubsub" -- you have to manually set up the script once, but it will run automatically when triggered.

If we're talking about protecting users still trying to learn the cloud, what if they mess up the setup of this cloud function?

It's worse – for us, budget alerts often arrive several hours after the fact, when you already might have burned lots of money.

Yes, well aware. I'm not sure if you actually looked at the code on that page but you have to catch the pub sub in the deployed cloud function in order to shut down billing on the project/account. This is precisely what I was referring to.

AWS has that too - you can trigger actions on budget alerts such as stopping activity https://aws.amazon.com/blogs/aws-cost-management/get-started...

What we need, a function that is AWS maintained and supported, that basically is AWS responsibility - that if it did not work - failed to shut down service that was consuming all the budget, they eat the cost.

This is simply false.

We go over our budget "limits" literally every month. We do this because they are simply alerts, nothing more. We have separate alerts set up for 25, 50, 75, and 100% of our "limit" so that we can track variable usage and catch spikes early.

We, of course, pay our full bill every month - over the "limit".

^this. For business-critical operations you almost _never_ want to have your service shut down, especially as a surprising result of a big spike in usage.

>future hires need to start somewhere, need to be able to learn on their own, and need to do so safely.

I've shared this before on hn, but I was about to bite the bullet on a fire base subscription to back a web app I'm building. Google announced its cancellation the week I was going to subscribe, I was hitting daily limits in my testing with my garage code and wanted the extra buffer to speed up my progress. As soon as they announced the cancellation I ripped out fire base and have been working on a swap to hasura/postgres/auth0. I can't afford, likely multiple, hundred or thousand dollar mistakes for a hobby.

"Just code better"

I'm trying...

Which clouds do this? I work quite a bit with AWS and GCP, and Azure to a lesser extent, and I've yet to see a spending limit on any of them. I'd love to be proven wrong though.

Azure definitely has subscription types that have payment limits. I have one subscription that has some credits on it, and if I go over the limits all services are suspended (i.e. compute, storage, etc) and I can't do anything unless I add more money or wait until the next period starts and the new credits kick in.

Not all Azure subscriptions types are like this though.

This would be the list: https://azure.microsoft.com/en-ca/support/legal/offer-detail...

Also documented at https://docs.microsoft.com/en-us/azure/cost-management-billi...

It's something that's frustrated me about Azure for years. They can do it, but they won't do it for pay-as-you-go accounts.

Make a SKU that's not for production with a $100 setup fee to prevent abuse and prepaid credits for consumption.

GCP has a spending limit, but it is by no means transparent[0]

Basically you add a budget to your project, create a pub/sub channel, then make a lambda that subscribes to the channel and turns off billing when it receives an overbudget message to shut everything down.

Or you could integrate it into your project directly to handle this more gracefully.

You can even send a test message, to check that everything does indeed shut down, but the whole process is extremely clunky and error-prone.

[0] - https://cloud.google.com/billing/docs/how-to/notify

Azure gives you credits to spend. And after you have exhausted your credits, then they don't charge more.

If you have Visual Studio professional or enterprise license, then you can avail recurring credits, which are useful for learning

Yeah, but that's $45 USD per month which is an expensive way to learn and test things that would normally cost <$5. It does get you $50 per month of Azure credits though, so it's probably the route I'd go if it's possible to buy a month of Visual Studio and ignore everything but the Azure credits.

I had (have?) an Azure account through my university account and while I agree that Azure services such as Container Service are (in my not so humble opinion) ridiculously overpriced but the fact it I never entered payment details to use my USD 100 of free credit on Azure. I effectively have spending cap of zero dollars.

I thought I was the only one who thought pricing on Azure seemed rather high for what you get. But I also have free credit I can use each month, which offsets the cost to "normal" levels, but can't imagine being a business and opting to pay for Azure services by choice, unless they provide something no one else does.

I worked at mega corp and we used azure services at >50% off the sticker price

Enterprise deals can end up changing the story it seems

That would explain allot. Totally forgot big business usually gets big discounts.

Had Bizspark from microsoft which gave us access to Azure and about $300 in credits I think per service or the whole thing iirc. They shut down services once we went over that limit. This was back in 2016/2017

With azure you can set spending alerts at the very least.

Why are people saying this? It's not true. App Engine had a spending limit, but it's deprecated and going away in July. The rest of GCP has never had a spending limit.

The best you can do is build your own service to monitor your bill and manually cut off your billing account if it gets too high. Hope you did it correctly because there's no good way to test it! And the bill notifications can be delayed an arbitrary amount of time with no guarantees, and disabling billing can take an arbitrary amount of time too, so you can still be screwed even if you did it correctly.

I think it's a glaring hole in cloud platforms. I get it, it's hard to implement in a way that is forgiving and doesn't cause a bunch of support tickets from people who set their limits too low and made their site go down. But that's why people use cloud platforms, so they don't have to implement the hard part. It's beyond me why the cloud platforms don't do something better here.

Why should they if they can use obscurity and people's limited comprehension of their complicated systems to produce legitimate, legal bills to customers?

You're talking a 'get paid/don't get paid' decision here.

When you look at this on a high enough distributed level, across enough users, this is the profit margin. Enough people can screw up and then be able to pay the cloud provider, that it is part of the business model. You're required to run some chance of producing a legitimate megabill.

Why be forgiving when your business model requires that you set traps for fools?

> As the article points out: Every other cloud does this!

No, they don’t, the article is misleading if not technically wrong (since they use “and/or” to link billing caps and better whole-project deletion, the latter of which mitigates a subset of the problem but doesn’t address accidental unbounded spend from a project that you want active at some level) and the same general class of complaint is raised for the other clouds on this issue; the frequency of the complaints seems to vary (ordinally, but not linearly) with usage of the various clouds.

Hmm not sure where the problem is. With IAM you can create a policy to limit the user to only use certain services, deploy only the cheapest type of instance type, limit the region, the time of day etc. All of this and more is built in to IAM by default. You can even create a CloudFormation as a internal product this way you can limit even more what people do, and budget for that. And on top of that you can make a Lambda that is triggered every 1h to shut down unused resources or send a warning to shut it down, if not, it will go down the next day.

The possibilities are endless. I personally don't see the IBM comparison.

Is it reasonable to expect a brand new aws user to know how to do those things?

Are you seriously suggesting that someone who wants to use AWS should go and read all the documentation and watch all the official videos available? That would take literally years. It's huge.

It also doesn't solve the problem. If the user makes a mistake they still lose a ton of money.

It's a really interesting problem. It's getting too big and too scary for a new developer to jump in and try AWS services, which means new devs will move on to newer services with better UX. I wonder if this sort of problem is one of the few existential threats AWS could actually face. Its possible it could end up as the Oracle or Salesforce of cloud services, very successful in enterprise but definitely not a good thing to have on your resume if you want to join a startup...

I would not want to be the competing cloud service, with self-imposed limits of only what people are willing and expecting to pay, going up against Amazon (or any other cloud service of that scale) where they not only have scale already, but are being paid the extra money regularly by people screwing up in small to medium ways (even IF the most massive screwups are 'forgiven' to devs who wouldn't have been able to pay anyway, but have twitter)

I don't believe there will be any new services with better billing UX through competitive pressure. You're basically asking the competing thing to take less money in order to win, as the distinguishing factor. It doesn't make sense as a way for them to win against the larger thing taking more money.

Probably the only practical solution would be arbitrary legislation forbidding the practice, as if it was anti-usury laws. Basically taking the guise of society and saying 'you may not do this, even if people are dumb'. And then you've got a problem (at scale) of people figuring out how to do hit-n-run megabilling and walking away under the legislation, having intentionally taken what they 'mistakenly' did. An 'oops, I accidentally a bitcoin' defense.

1. yes I am, studying years to be a doctor, lawyer, Engineer etc from your point of view is also to much? Should you just start cutting people to figure out how a body works end expect for everything to be ok once you are done? The point being, yes you have to study to learn something, there is no way around it.

2. If money is a concern to you, then I should focus on learning how exactly the billing works and how to monitor correctly. This way you can build a product the right way, not to mention AWS by default has limits on their services set with limits that prevent you from doing something incorrectly. For example you can only make 5000 requests a sec on the API Gateway, you can only have 1000 concurrent lambdas, you can spin only 25 ec2 instances, ecc... (true, not all services have limits like this - but then again, if you want to use one, the first thing you should do is check the pricing page, this is what I do fro every new service that I'm planing to use).

3. AWS is not for developers, it is meant for SysAdmins and DevOps (true that some marketing materials are not clear on this), they should be the one configuring it to allow developers to host their code. If you want a turn key solution, then there are better solutions, like Heroku - incredibly easy to use and understand and have a much simpler billing structure.

With AWS you can do anything you want, AWS provides lego blocks, what do you build with it is up to your imaginations, and for sure it is not meant to be use directly by developers who have no idea how networks, computers, databases, cpu, ram, policies, storage etc works. Developers should focus on coding, and SysAdmins and DevOps should focus on managing the infrastructure.

And if you want to learn AWS because you want to be a SysAdmin, then it is true, that AWS could have a plan for beginners with even smaller default limits, and limits set on everything - this way you could more safely play with what they have. This would be a nice things to have in this case for sure.

But because they don't provide such thing, you need to be the responsible one, and start learning AWS the right way, and not get in gun blazing, and expect all will be ok. My recommendation is to learn one service at the time. If you do this, over the years the acquired knowledge will be gold. Plus the more services you learn the right way the easier it gets.

There’s room for a little bit of the personal responsibility argument, but it’s ineffective when you take it way too far.

> studying years to be a doctor [...] Should you just start cutting people

Spoken like someone who’s never used AWS or been a doctor. Your analogy is horribly, badly flawed. Doctors do start cutting people, in the US nearly all med students dissect a cadaver at the start of their first year. What you’re suggesting is doing years of pure documentation reading, unguided by other people or a curriculum, before practicing AWS, which would be silly and a waste of time. People learn by practicing, which is why med students dissect cadavers, and which is why AWS offers a “free” platform to learn by practicing, tutorials to guide the learner, and advertising to attract learners.

> If money is a concern

Maybe this is why AWS offers a “Free Tier”? https://aws.amazon.com/free/

> AWS is not for developers

That’s not what AWS says https://aws.amazon.com/developer

> Maybe this is why AWS offers a “Free Tier”? https://aws.amazon.com/free/

People in this discussion are complaining that the free tire is misleading, plus not all services are covered. And it is true that if you turn on a bunch of server and you don't pay for a year, but forget about them, you will be charged the moment the year passes. Not to mention that you will be charged if you use the CPU of the free tire server to much - which probably very few people know about.

> That’s not what AWS says https://aws.amazon.com/developer

And my point is that the marketing of AWS is misleading, they try to convince you that if you don't know anything about computers, but you know code, you will be able to manage AWS. This is very misleading because AWS tries to make you think that AWS is a service like Heroku, simple to use, and there is just one button to push to make it all work. Completely false. I've seen countless AWS accounts that were completely misconfigured by developers who thought that AWS easy to manage. A basic example is the autoscaling of EC2. People will go to the autoscaling section of EC2 "enable it" and be superseded when it dose not work. Where the reality is that you have to do 8 other things to make it work, not to mention the work that needs to be done in the OS itself.

What do you think about this page: https://aws.amazon.com/getting-started/?

> Select a learning path for step-by-step tutorials to get you up and running in less than an hour.

Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?

> What do you think about this page: https://aws.amazon.com/getting-started/

It is very dangerous. If you select the full-stack tutorial you get: "Time to Complete 30 minutes". It should say: "30 min to ruin your life" ;)

If you want to really learn AWS, then this page should be used as a reference of how to design a stack. If I were you I would read the tutorials to see which services are needed for a solution, but before doing anything, I would read the docs for each of those services to really understand them, then I would go back to the tutorial and actually do it, and - MOST IMPORTANTLY - I would read the pricing page for each service that you are going to use.

> Do you think it's irresponsible for AWS to encourage beginners to try their service when they apparently only intend it to be used by those with a computer science degree and 5-year apprenticeship under an experienced sysadmin?

100% - when I started working with AWS in 2016 I had a very hard time figuring it out, because I was looking for the simplicity the the marketing team was writing about. I really don't like what the marketing team tries to tell you, because it dose not exist.

Regarding an approach to learn about AWS, I would start with all the serverless services that they have, since the pricing for most of them is ideal for beginners (WARNING - read the pricing page for each since not all have a free staring plane, like S3 and DynamoDB) and for simple weekend projects.

For example, I did build this project a while ago: https://github.com/0x4447/0x4447_product_s3_email, if you scroll down to the pricing section you will see this:

``` All resources deployed via this stack will potentially cost you money. But you'd have to do the following for this to happen:

- Invoke Lambdas over 1,000,000 times a month - Send and receive over 1000 emails a month - Perform over 10,000 Get and Put operations and over 2000 Delete operations in your S3 Bucket - Exceed 100 build minutes on CodeBuild - $1 per active CodePipeline (must run at least once a month to be considered active)

The only payment you'll encounter from Day One is an S3 storage fee for emails and CodePipeline artifacts. ```

So you can have a stack that is actually doing something very useful that costs not even a $1 a month.

It is possible to pay $0 to AWS, but you need to first understand AWS to be able to do it, another trivial example of a tiny project that is useful and cost $0 to run: https://github.com/0x4447/0x4447_product_secure301

The last point would be: don't listen to the marketing material - they are there to sell you AWS, marketing never cares about reality.

I also recommend this website https://awsvideocatalog.com - pick a service and watch all the keynotes AWS has on that service, if you'd spend 1h a day, in 6 months you'll know more about AWS then anyone else complaining here.

Agree with all of this. I think the reason you're getting so much pushback is that you started this conversation with "not sure where the problem is". Clearly you do see the problem - AWS encourages beginners to get going as quickly as possible with these tutorials and the free tier, but then makes it difficult for those users to avoid unexpected charges. They should either stop encouraging beginners, or start offering easy ways to protect yourself.

That statement was related to the sentiment that there is no way to protect yourself or a team of people from AWS pricing, people were implying that there is no tool to help you limit or track expenses, which is not true, since there are plenty of tools to do that.

Anyway, life gose on, and it was overall a good chat :)

> I need to stop blaming others and start owning my mistakes.

There's a big difference between owning up to your mistakes and taking on a bunch of unnecessary risk in situations where you know it's more likely you're going to make mistakes. IE: Learning, testing, etc..

In fact taking on unnecessary, possibly unlimited financial risk in a situation like that is a mistake. Want to own that one?

You can by insure for you as a driver and for your car (and in many jurisdictions have to) — this limits your responsibility. You can spend $10000 on AWS and still make a mistake and pay another $10k+.

I don’t get the point that AWS docs are free. I mean, they are free, but the services are not. Do you pay for a visit in a car dealership?

> You can by insure for you as a driver and for your car (and in many jurisdictions have to) — this limits your responsibility

Real insurance policies have maximum coverage limits, so, while it reduces your liability by a capped amount, it doesn’t, strictly speaking, limit it; you still face unbounded potential liability beyond your insurance coverage.

what's the biggest unexpected billing of AWS fees reported? I'm thinking 100,000 seems unlikely to be reached very often.

on edit: this is not to say that I believe an insurance for this kind of thing would really work (although maybe that is because I know nothing about the insurance business) just that I don't think the problem with the insurance would be that you could always end up getting billed more than you were insured for.

This really is not true. Spending limits are not supported by all major cloud providers.

It seems obvious that AWS simply doesn't want you as a customer if you need to implement spending limits. We've all talked about having customers that were too much trouble for the value they bring. Customers with hard spending limits are those customers for Amazon.

So items 1 and 4 are courses which you need to sign up for and get everything including instruction. That fails to qualify as "on their own".

Item 2 is documentation. This is good! But without access to the system to try it out, how self-teachable is it?

Item 3 is the one counter point. With a third party emulator though, how comprehensive/accurate is it? Would you hire someone to work on a mainframe who'd never used a real system before?

Compare that to e.g. Java. Sure there are plenty of college courses, some even Oracle affiliated, but I can also just download a Java compiler and start plugging away. .NET is there with Visual Studio Express. Oracle cloud, for everything else wrong with Oracle, has a free tier with no time limit. Microcontrollers for embedded development can be had for €20. You can install Linux or a Linux VM on your own computer. Even stuff that falls into the more enterprisey camp like MS SQL or Oracle DB has free personal licenses.

>So items 1 and 4 are courses which you need to sign up for and get everything including instruction. That fails to qualify as "on their own".

Attending a course on your Own.

>Item 2

No it's a MVS distribution (a full blown Operating System) with all the compilers 99% the same as a "modern" z/os

>Item 3

That's the S/390 emulator.

>Would you hire someone to work on a mainframe who'd never used a real system before?

Most Mainframe-Programmer where never near the Hardware anyway...so YES i would do that absolutely (you don't do Hardware near programming in Mainframes, an emulator is perfectly fine), same as a Dev who programs in a VirtualBox, no difference.

If you want to learn how to use it, and MUCH more:

https://www.youtube.com/channel/UCR1ajTWGiUtiAv8X-hpBY7w

https://github.com/moshix

>Compare that to e.g. Java.

You can do exactly the same with MVS, all the compilers (and more as z/os brings with) are in it.

My business uses off-cloud backups, devops and prepaid credit cards to limit our risk exposure.

Forgive me if I am out of line, but as I understand it, your company would still be legally liable for the debt. So even if blew past your prepaid card limit, your company would still have an obligation to pay using an alternative method.

Not a lawyer, etc, and don't even know what jurisdiction you are in or what providers you use, so this is just a general caution that your strategy may not generalize to other companies.

I assume you're not also using AWS? Otherwise even with a prepaid card, on AWS you are granting them blank checks via unlimited invoicing abilities.

I’m not sure I’d want a running business haemorrhaging $10,000/minute or whatever is possible for cloud compute.

My business is looking at cloud compute at the moment and we have absolutely no idea how to do it safely. In fact sagemaker is exactly a product we have looked at and discounted since we cannot be sure we can do it safely without getting unexpected megacharges.

We had an incident recently where a cellular router was set up to send a text message on a GPIO changing. The old firmware didn’t have pull-ups on the input. We told our end customer to update the firmware. They didn’t and also didn’t hook up the input and left it floating. We got a £20,000 bill for text messages as a result and a soured relationship with the customer.

What if that engineer fat fingered and reserved a million GPU instances instead though?

It would be nice if cloud providers were better at surfacing potentially company killing things via their dashboards and nag emails.

Well, they’d have to first contact support and get the instance limits lifted… it would be one hell of a fat finger.

Not everything in AWS is a VM with a limit.

For example, their CDN has no "TB per month limit". If your site gets popular and is 100% cacheable, you could be getting a million dollar bill.

I hear similar horror stories about the various distributed databases in several clouds. They're all crazy expensive.

Great: Now do the same for everyone. Let everyone sign up for an AWS educate account with limits they choose.

AWS employs cost obfuscation by design otherwise the default view when you open the console would show you all of your current active services. Not only is that not the case, a single screen to show you all of your current active services doesn't exist. You need to take a deep dive into cost explorer (assuming you have access in corporate land) and try to decipher in what that all means.

They definitely need a senior executive to stand up and say, "The Customer wants us to be transparent in billing, fix that now."

Then they need to start a team dedicated to finding a good way to let customers halt spending at a given limit with minimal impact on their operations.

They already win on UX (okay, okay, it's an opinion ffs), but unlimited liability makes a lot of people very uncomfortable. Those two actions would go a long ways towards demonstrating good faith in that area.

If it would cost too much, maybe they could present it as an easy way to cut expenses at the same time that they introduce a small price increase. This is a common and long-standing complaint/feature request.

"They already win on UX"

As someone that has tried and failed to get some small personal sites running on AWS a couple times, I'm going to have to tag this snippet with [citation needed].

From my admittedly limited experience with GCP and Azure (and this is obviously subjective), the UX in the most successful competition is, at best, no better than AWS. It isn't that AWS has good UX, just that all the cloud providers have bad UX.

Oh man, I have long been joking about the AWS UI. Like how it will gladly walk you through all the steps for launching a server and only at the very last step says “oh you don’t have permission to do that lol, get permission and start over”.

I compared[1] it to a bartender who walks you through an entire sale and only at the last second rejects your purchase for being underage (instead of denying you the at the initial request for something alcoholic).

Of course, that’s one of the minor things in the grand scheme.

[1] http://blog.tyrannyofthemouse.com/2016/02/some-of-my-geeky-t...

agreed especially when compared to say Heroku or Digital ocean. I see many new comers struggling to deploy a small website. It's overwhelming. I understand that Heroku exists for such users and they are using AWS under the hood, but is there any service which takes AWS cloud APIs and simplifies it with a leaner UX?

> agreed especially when compared to say Heroku or Digital ocean.

Funnily enough, I've never ever been able to understand what Heroku is, or how to deploy anything useful on it :) But I'm old, and I've always found it easier to tinker with nginx configs.

Heroku is very simple - you just create an app and "push" code to a repository tied to the app. You need to write code of course, but they auto-detect the language and prepare the environment or platform for you (so platform as a service). They tried to appease the geeks by making everything powered by CLI - but otherwise its just point and click for deploying apps.

Tinkering with nginx is good, but what if you want to add a database, a cache, a logging service? What if you want to automatically build code when you push to github? What if you want to easy clone multiple environments (for staging, prod etc) without never doing an SSH into server? What if you want to have versioned releases that you can easily rollback to a specified version or push a branch into one environment and another branch into another environment? And you get a workable subdomain for every environment you build with SSL enabled. All of that can be scaled down and up with literally few clicks.

> but what if you want to add a database, a cache, a logging service?

I... just add it. How do I add it in heroku where everything is "just create an app and push it" and everything is separate "dynos" (whatever that is), each with different pricing, storage, bandwidth etc.

that is true. everything is a separate dyno - means separate machine/vm to which you need to connect to. You get free tier for almost all databases and you only need to know how to connect to it, not tinkering with the storage, tune the parameters, scheduled backups, automatic scalability etc - that's done for you. it will become expensive once you go past free tier or the cheap tiers though.

> not tinkering with the storage, tune the parameters, scheduled backups, automatic scalability etc - that's done for you.

Thing is: if we're looking for free, that may make sense. you still need tinkering though: you need to figure out what the whole dyno thing is about and how to connect from one dyno to another, and so on.

However, if we expand this a tiny bit to cheap and/or free, then simply running the lowest-tier server at Digital Ocean will be a better proposition. And installing a database these days is basically the same thing: run a command to install, all you need is to connect to it. No tinkering. And it will still probably scale significantly better than a free tier at Heroku :)

Once again, I'm biased and old, and can afford to spend money on side projects. I know that sometimes even $15 a month is out of reach (oh, I've been there). But yeah, this sorta prevents me from ever understanding Heroku :)

I think that's basically what Lightsail is supposed to be. I haven't used it myself, but from glancing at it they're pretty clearly targeting DO/Linode/Vultr etc.

I think there's some verticals - players that simplify/aggregate SES, or S3, or whatever - but none that handle being a layer on top of most/all of AWS.

AWS have a multitude of services that do this already. I’d recommend to go through their services one by one and learn what they offer.

It’s not you. It’s pretty rough for new users.

Have to agree with the others. For example, while setting up ELB it's possible to select at least one option (un-checking the Public IP box) that causes the setup to just fail with a nonsensical error message. Turns out ELB requires a public IP to communicate with the nodes. That's just the most glaring one off the top of my head.

I also remember trying to set up SFTP whenever it was first released. It was literally impossible to do what they were advertising (S3-backed SFTP with jailed homes for each user) without writing a manual json config (I never got it to work). I had built my own solution for this exact thing on EC2, using a bash script more or less, and thought the hosted option would be less of a maintenance burden. Needless to say I quickly gave up.

Unless you understand IAM, the scoping of individual users from the sftp service can be a bit confusing. What you need is a scope down policy:

https://docs.aws.amazon.com/transfer/latest/userguide/scope-...

The more annoying part is that the service only supports public/private key logins. If you want user/pass you have to write a lambda. The lambda is pretty simple though, it checks the credentials (so it can hit any backend you like), and if they pass it returns a 200 with a json doc of role (which is just sftp assume role), policy (the scope down from above), and home dir.

https://aws.amazon.com/blogs/storage/enable-password-authent...

This touches on a larger issue with AWS though. It's that they are trending to leave out functionality and point to lambda as the solution. On one hand, I get it. The lambda solution is infinitely more flexible, but what if I just wanted an sftp for a couple users that uses user/pass?

To your final point, it is so much less maintenance. There is no server to manage, and since I want all the data in s3 anyway, it's already there. This solution replaced a server with chroots, EBS, scripts to move data to s3, etc...

How are people not building in Terraform for an easy ‘destroy’ at the end?

I know it’s rhetorical and a lesson learned myself, but yeah… I would expect folks to learn to use this tool to help manage costs this way.

terraform 'destroy' isn't infallible. There are certain resources that trigger the creation of other resources (for example, lambda functions will create cloudwatch log groups, and dynamodb tables create cloudwatch alarms) and when terraform destroys the resource, it doesn't necessarily clean up all the associated resources.

Terraform has it's shortcomings for sure, but I don't think it's reasonable to expect Terraform to go out and clean up second-order effects of it's resources.

I'm not doubting that the situations you describe are true, but abandoning resources like that is an AWS-lifecycle problem, not really a Terraform one.

Sure. My point is just that `terraform destroy` doesn't necessarily solve the problem at hand. And you could still end up continue paying for those second-order effects after running a terraform destroy.

I am pretty senior and can thus sometimes afford to do a new thing and try doing it the right way, at the same time. Not even always. Many people just take one learning curve at a time...

Typically Terraform takes longer to get something working than mindlessly clicking through the console. In my experience those mindless clickthrough things end up sticking around for years even when they weren't intended to.

This is why you have separate development and production accounts: a development account where you mindlessly click through so that you can learn through the UI what's available and how it works; cleaned up on a regular basis by something like aws-nuke, and a production account where you have the discipline to only create resources through Terraform / CloudFormation etc.

cdk is promising for things like this, way safer and easier than rolling your own scripts with bash or python.

Even CDK sucks though because if you’re still kinda new to it all, you want to login and make sure that it’s all hooked together correctly. And you’re back using their shitty UI.

Why you can’t look at the load balancer, look at the listener and then show what’s in the target groups is beyond me.

I have a different take on it. I started out doing everything through the console, then learned the cli and boto3, and very recently CDK.

CDK is another tool that build on the cli and boto3 concepts, and also manages the orchestration and dependencies.

Having to go back to the console isn't a fault of CDK. Learning the right tool to use for a situation is part of the learning curve. I go back to the console all the time to look something up quick or to understand what config options are available. Or I repeat the same steps in the console enough times that I get bored with it and automate

Edit: also I tried and have given up on CloudFormation more than once. CDK is like a wrapper layer around it, and has been pleasant to use.

I’m not trying to be snarky here, would like an honest opinion: how do they win on UX? Are there specific things you like there?

I’ve used all three major clouds in production now and I dread using the AWS console. Or really any part of it, over Azure or GCP.

I’ve always thought of them as purely winning the “nobody got fired for…” mindshare thing despite having a thoroughly mediocre product.

It either works great or barely depending on the service you're using - some AWS teams have dedicated dashboard teams, eg 'ec2 dashboard team' which solely focus on the dashboard experience, while others touch it as an afterthought.

I'm pretty sure something along the lines of this^ was posted on HN by a former AWS employee but I can't find it now.

Their web console UIs vary widely by service. IMO, the UX for most of their simpler services, specifically SQS, S3, Lambda, DynamoDB, are really easy to use and work nicely. If you want to start with a Docker image, send it to AWS, and have it get spun up and attached to a DNS, well that's a huge complex mess to get set up.

Unless you use something like Terraform or Pulumi, in which case it’s almost a one liner.

The AWS default view feels more crisp for me, compared to the GCP one which feels very "floaty".

Their UI toolkit for that at least is on point.

This gives the other vendors a wedge in.

I don't understand why so many people want to use AWS for such small budgets. At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?

AWS is awesome when you have a large number of resources, that are created programmatically and reproducibly, with redundancy and duplicate environments.

The budgeting tools are really amazing at letting you categorize your costs and create appropriate alerts. The permissions system lets you define very specific roles.

Its complex. But if your system is complex, it gives you the tools to keep track of it all. If you have a <=$5000/month budget, it is probably too small to make sense in AWS. You can probably run your system on a couple servers.

> At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?

I have a handful of AWS Lambda functions with a DynamoDB backend serving hundreds of clients, my bill for the month of April was $0.01.

No, a VM wouldn't cut it.

But you are right that there are certain slots where AWS doesn't make sense: There's one in the lower middle range where you can save a bunch of money by using a VM or two with your own DB servers. And there's the one where you're so big it might actually be worth it to implement the whole stack yourself.

> I have a handful of AWS Lambda functions with a DynamoDB backend serving hundreds of clients, my bill for the month of April was $0.01.

What kind of thing do they serve? Somewhere I could read more about this kind of project?

It's a common backend for chat bots (Discord, Matrix, IRC, Telegram).

Basically it takes different inputs and commands and uses different APIs to fetch data based on the input efficiently without the need for web scraping. DynamoDB is used mostly as a cache for common queries so I don't go over API quotas.

Most of the bots are made by me, but with AWS API Gateway I can easily generate API keys for anyone else and keep track of their usage.

This is considered a cloud native pattern.

https://docs.aws.amazon.com/apigateway/latest/developerguide...

> At that scale wouldn't it be easier to just build everything out of a few VMs at a cheap service?

No. I looked into AWS vs Azure vs Cloudflare vs Digital Ocean for something I'd like to build this year and (for my thing) Cloudflare (Workers) was the best balance of cost, scalability, and maintainability.

> I don't understand why so many people want to use AWS for such small budgets.

Serverless (I hate the term) makes a lot of sense for small budgets and projects. You're investing very little, so the downside of (severe) vendor lock-in is somewhat low. The upside is HA infrastructure, horizontal scalability, zero maintenance, ignorable monitoring (at least initially), consumption based costs, etc..

The biggest problem with all of them, except Digital Ocean, is that it's difficult to "own" your data in the sense that you can download a copy of a DB and keep using it somewhere else. IIRC Digital Ocean has a fairly nice managed PostgreSQL offering, but it still scales similar to a traditional DB (ie: not automatically).

The next biggest problem with AWS and Azure is that you can't figure out what anything costs, at least not easily. For example, I know for a page I'd want to serve via Cloudflare Workers, I could do 1 hit = 1 point read from Azure CosmosDB, but I couldn't figure out if the pricing includes egress. Just look at the pricing page for CosmosDB [1]. It's ridiculously complicated and that's one service on Azure.

1. https://azure.microsoft.com/en-us/pricing/details/cosmos-db/...

Linux server administration requires a certain amount of know-how and determination. I can't tell you how many times I had to rebuild a DigitalOcean server because I messed something up and wanted to start with a clean slate.

A lot of hackathons, workshops and courses ask you to use AWS these days. Whether it's to run machine learning instances, win a sponsor prize or learn how to use Lambda, students are often encouraged to learn one of the major cloud providers.

Also it's a resume boost. It's another buzzword you can add to your resume.

Disclosure: I'm the Co-Founder of Vantage.

This is exactly what we do with Vantage: http://vantage.sh/

We give point-in-time run-rates of all active resources based off of the region and resource/service configuration.

In addition we try to simplify people's understanding of where their costs are coming from. If anyone needs help with this, they can personally reach out to me at ben@vantage.sh

With all due respect, I think the issue here is that your company should not need to exist. Amazon should provide this feature as part of the UX.

Yea they should, but if it doesn't, are you gonna keep waiting or do something about it?

I would be more worried that if you do something about it, Amazon is gonna do something about you.

Starting with obnoxious API changes, followed by ToS changes. Hopefully that's as far as it has to go.

Agreed. Nothing new about this request.

It's like hiring a plane to fly over your house repeatedly to tell you whether or not it's on fire. There has to be a better way.

Same respect, nice tool, but it is sad it is needed. But I guess it's similar to lawyers, police, army, I don't like the need for them, it, but good they exist.

Azure does this a little better, but best would be to see a breakdown on the invoice with links directly back to the resource.

Maybe there are discounts or other processing that makes this hard.

Or, less charitably, this would lead to people optimising their costs a lot better and canceling unused services much sooner.

Azure groups most things into resource groups which greatly simplifies things.

Yes. Also split out into sub categories. How much of an ec2 spend was on bandwidth and how much on compute for example.

Enterprise Agreement accounts aren’t billed on demand, so there’s little use for that in accounts spending a lot of money.

Why the f do I have to use a service like billgist to get a breakdown of my services?

AWS billing IS complicated but could be made so much simpler...

I'll bite...

Because, if you are a company that uses AWS at scale (deploying literally thousands of resources at a time), you care more about meeting demand and getting resources to spec, than you do about the cost of an individual elastic IP...

The price of each individual resource isn't something that you want to see on every screen you touch. It literally clutters the console with information that you couldn't give a shit about when your company is making millions (or billions) on the services you provide. You care more about your service's reliability/scalability/uptime/etc than anything else. This is priority numero uno.

If and when cost analysis becomes priority, you look to see where you may be overprovisioning resources - hence, the billing console.

But for the larger players on AWS (the multi-billion dollar ones that AWS cares more about making happy than you or I), an extra $100k in AWS expenses in a year isn't a worry, it's a write-off.

You also get much better negotiated rates for everything when you are big.

I agree, having set up serverless and lambda for an api/app that was to be used a few times per day, the billing made no sense at all, it would increase even though the services were used less and was difficult to find what the costs were or whether they could be reduced, eventually I had to shut it down and create a non-lambda solution somewhere else because I had little control of the cost.

If I go to billing I can see an itemized breakdown of every service I'm using. What is wrong with that?

Students and self-learners want to experiment with doing complicated things on AWS without getting a surprise $3,000 bill because their script didn't shut down those instances like they thought it did. They want a hard fail-safe to protect them from surprise bills. And they want a solution that's easier and more reliable than checking the AWS itemized breakdown twice a day.

I agree with that. That is why there are "Billing Alarms" in big letters on the billing page. But that wasn't the point of the reply: the person above me was complaining that there's no way to tell what services are being billed, which is bunk.

The breakdown isn't granular enough by far. ec2 is a very big bucket to go digging into for costs.

This is what keeps me from tinkering with the free tiers and never even attempt to host my own projects in the cloud.

I want a free tier with a lock on it. I'd very much appreciate advice for what tiers would fit my monthly use if I hit free tier caps, but if I got hit with $500 right now I'd be ruined.

I think my Google compute stuff is safe, but I really don't like having any doubt.

Until AWS fixes this the best thing to do is just use their service as little as possible. There are plenty of other cloud providers out there these days which don't employ this hostile practice.

I closed an AWS account for this reason just a few days ago, we hadn't used it for a while but there was no clear way to remove our credit card so it felt like a risk just to have it open, what if a developer logs in to mess around and accidentally flips some switch that smacks us with a charge of a couple grand... unlikely, sure, but the fact that it's even possible is terrible system design. Better to just nuke the account and move on to other cloud providers that don't make it harder for me to sleep at night.

Not disagreeing with you (a view with all the active services would be great) but one of the many benefits of using Terraform is that it allows you to know what you running.

Yep and is a huge SaaS market place now so mid to large size companies can figure out what they are spending money on.

I don’t use them for any projects mostly because I already don’t enjoy having to manage AWS at work all the time. I also don’t want to live/work in a world where AWS is my only option so I try and use smaller hosting providers and services like tarsnap.

I cancelled an account 2 years ago because no matter how much I explored cost manager or any regions I used, I can't find why I still get billed a small dollar or so a month.

I still get bills on the 3rd of each month, followed by a "we can't charge your card" (I removed it when they pulled this shit) followed by a few days later you get the "hey we're gonna suspend your account if you don't pay"

Yeah cool, I only closed the account 2 years ago now, go right ahead and suspend/close/do whatever you've gotta do, I stopped caring.

The cost explorer is beyond deplorable. It provides either too little information or information that is so fine-grained it becomes useless.

It infuriates me the hoops we need to jump through just to understand how much a single virtual machine costs. Or in cases like yours - to find out what resource a charge was for. Impossible.

Designed to be impossible. FFS. We would spend more money if we were able to manage costs better.

Same! I literally closed an account and created a new one after a few months because I was charged a few bucks a month and I didn't know what I was paying for.

I think it was either related to db backup or some encryption keys but I couldnt find what and where to delete to get rid of this charge.

Nowadays I use aws only for route53 and s3.

Same here. I had a monthly billing of a few dollars and couldn't figure out where it's coming from (and given that it's only a few dollars, it wouldn't make sense to spend many hours investigating). In the end I just canceled the card and hoped for the best.

Came in to say I had to do the same thing, close i.e an account to stop being charged.

Yup. Got charged over $800 for "experimenting" with a DynamoDB database and forgetting to delete it afterwards.

Sure, I called customer support and they reversed the charges. But something the nice lady on the other end said as she chuckled: "This happens all the time".

DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.

Apparently I owe AWS 1 cent for DNS. The problem is I can't login to pay it, so every month I get a email that says "your aws account is going to be suspended" and 30 days later I'm disappointed that they didn't follow through with the threat.

AWS charges me $1 or $2 every month, and I log into my account and click through every page and can't find a single active service or any clue on what it's for. It's marked "EC2 - Other", whatever the f that means

Yeah, that's an elastic IP, check under EC2.

People forget to clean them up after shutting down the instance (probably kept in case you want to keep your IP)

It's crazy that it's so common, but that they wouldn't make that obvious.

Considering the whole IPv4 extinction that's coming down and all.

So an elastic IP is just an AWS provided IP that they hand over to you... it's free while your instance is running but if you stop your instance, you get a small bill (it's like $0.84 a month)

I will gladly welcome the eventual IPv4 extinction, but unfortunately I'm pretty confident it won't happen for another 30 years.

I don't have any elastic IPs. The only thing under EC2 is a security group that it won't let me delete, but seems like no resources. My bill this month is $2.88 /shrug

Check volumes, non-root volumes are kept on deletion but should be EBS charged instead.

What does the bill details screen show? I manage an AWS bill and I find it overly detailed. Others mentioned leftover EIPs, but the bill shows these as their own line item.

Not saying it can't happen, but I've never seen an 'Other' listed on the bill details screen. As I said above, it's detailed to a fault.

Actually it looks like this is for an EBS snapshot. Now it seems like EBS is a Lightsail feature, and the Lightsail console says that I don't have any storage or snapshots? Hm

EBS is also the general external storage attached to an instance. Go the ec2 console and look at snapshots and you should see it there. They are region specific, so make sure to select the region referred to on the bill.

I haven't used Lightsail, so I'm not sure if those snapshots end up in the general ec2 console. But, if you already checked the LS console and didn't see anything, then they must be in the ec2 console.

Possibly a disassociated elastic IP address?

I wonder if there is a human whose job is to final review suspensions and they just keep shaking their head at the measly pending $0.01 charge.

I keep a penny on my desk at work. Every time a coworker comes across somebody's account that is off by a penny, I offer it. They think I'm being funny, but the point I'm trying to make is that it wastes more than pennies worth of our time to spend it worrying about a few cents.

I've seen people mail in checks worth less than the stamp it took to send them. It's a wild world out there.

The interesting thing is, accounts being off by a penny or two (excluding actions such as customers doing a mis-type in the bank transfer) can actually point to deeper issues such as improper rounding somewhere in the path.

Usually they wrote us a check for the wrong amount at some point and end up with a small balance remaining in their account. We don't want to just forgive and forget every time somebody owes us $0.16, but if it's been there months and it's going to take extra effort to get it from them, it's probably not worth it.

We do try to determine where the discrepancy started and watch for indicators of a bigger issue.

Or a compromised system https://en.wikipedia.org/wiki/The_Cuckoo%27s_Egg_(book)

Hahah yeah I say the same thing every month "yeah okay go ahead and do it then"

I have to imagine that these things are manual and a human has to do it, so they'll sort it by the size of the account and do what matters

> DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there.

Did you look at the PDF invoice in the Billing section? I have never seen an invoice where the line items didn't add up to the amount charged to my credit card. If there's just $2.40 on there with no explanation, I'd open a support ticket to complain.

(While looking into this, I was surprised at how minimal DO invoices are, however. For GCP, I'm used to seeing on the order of a million line items per month. Seeing only 3 on my DO invoice was surprising, and could definitely lead to a case where something isn't accounted for correctly. But, I bet support will fix it up for you.)

> DigitalOcean is the worst with the dormant accounts. Just got dinged around $2.40 on my credit card. Going into DO I could not find what was causing that charge. There was nothing there. Wuuuttttt.

Oh I used to get this, for me it was snapshots of a VPS I destroyed. Completely my fault, but yeah it was annoying to figure out at the time. Probably worth checking you don't have any reserved IPs, disk backups or snapshots left over

Honestly I get lost inside of AWS. Only recently was I able to figure out why I was getting charged $.82/month which, in the long run, is really nothing. But it's amazing how hard it was to figure out why I was getting charged for something that I originally thought was just going to be free.

What did it turn out to be that was charging that?

I used the AWS free plan many many years ago as a student. I was so paranoid about losing money that I immediately shut what I was using down after I thought I didn't need it any more (even though leaving it on was part of the free plan). I turned off the instance, but I forgot the public IP. A month later I got charted a few pennies because an associated public IP is part of the free plan, but an un-associated one isn't.

I didn't go bankrupt, but it proved to me how scummy AWS can be given that actually trying to use less is what got me charged.

I have a DO server thats been running for more rhan 5 years. It was the basic $5 bucks instance they offered at the time I felt a bit robbed a couple of days ago when I was spinning another instance and I saw that the newer $5 instance has more memory and more cpu power. I feel like they should have decreases the cost of the older less performant instance.

I have been spoiled by my Internet provider who year after year they will automatically increase my speed as they made it cheaper, as I paid the same amount for the service.

When they changed the pricing they emailed me and told me that I could effectively reprovision my machine to get the price difference, I think it was part of a rollout of some other infra as well.

I think I'm going to save this whole thread as "why zero friction micropayments are never going to happen".

The billing statements I see in DO under Billing (sidebar) > Billing history seem pretty verbose to me. If it's not there I would recommend reaching out to their support.

Ugh. I'm so glad my first owie was "only" $200. Hurt a lot at the time, though. I could still use that back...

Isn’t DO some random saved image from years ago? I’m still paying them for a few of those I think.

Yeah but it shows they are reasonable and flexible if a customer makes a mistake. The reason Google Cloud is third, people remember how they were treated when they used Google for Work or whatever its called now.

> But something the nice lady on the other end said as she chuckled: "This happens all the time".

Is better then if they said: "We reminded you about the consequences of not deleting, we have done nothing wrong."

I would not go that far to say it's fraud. Calm down.

Maybe it's something I missed. Maybe its some hidden feature that I did not turn off. But I deleted all my droplets, all my IP's all my firewalls, etc, etc and could not find anything else.

You have in good faith attempted diligently to have a zero purchase. You cannot even find any purchases. In that case they have reached into your pocket and taken money without your consent. That is just a fact. By mistake? It's being short changed in a shop at the absolute best.

This is not a mistake any company should be able to afford to make. Ever. Under any circumstances. If it really is "just an error" they need to making amends in a very public and obvious way that is clearly expensive to them to show good faith.

But we don't get that. We have to fight from a position of being in the wrong when we have money taken from us by these fraudulent practices to overturn them.

It is necessary to go HARD at the morality of "accidental theft" because it is so prevalent and because all these companies clearly don't consider it a detriment to their reputation.

You've been diligent. When does it become their responsibility to not reach into your pocket when you clearly don't want that. Or anyone's pocket. I'd bet quite a sum that where there is one there is many.

This is where I'd say that every single paid service needs to have a way to contact a human being, and any cancellations expressed to that person by an authenticated user must be respected. If services are allowed to construct Gordian knots of byzantine interacting services, then there must be a way to cut through that mess.

Perhaps you have a snapshot?

You should be able to look under your bill for a breakdown on what service the charge is for.

I'd double check that you don't have multiple orgs. You used to share a cess to your account with people, then they made some org change a while back that essentially moved that shared access setup to an org and gave you a new personal account iirc. Easy to not realize you have both.

The invoices on the billing page break down where the charges came from. If you download the pdf version, it has even more details.

Could it be a convenience charge?

Perhaps a billing fee?

> “It’s the student’s responsibility to know what they’re deploying.”

Anyone who seriously argues this is 100% unaware of the quagmire that is AWS billing. There are companies with entire teams built around just optimizing AWS billing - it's whole unsurprising that some AWS feature actually spun up 5 separate AWS features that end up being billed.

Not to mention it only goes one-way. If you destroy the service that created all those others, it won't clean those others up.

And even when you spin things up knowing their cost you may not be accounting for data transfer, storage, mismatched reserved instance purchases, unused EIPs, and so many more "hidden costs"

Ill seriously argue this.

The issue is that when students are "learning" "cloud", they are not really learning cloud as much as following tutorials to click together turnkey solutions. Dynamo DB, ElasicSearch, and whatever else are all business-oriented, quick to set up services. And such, these services are hiding what is going on underneath, with the expectation that the company will cover the costs of whatever those services run. This is a standard buisness practice.

If you want to learn how to put together cloud solutions with those technologies involved, you don't even need an AWS account. All you need is a decent laptop or a desktop, with docker or VM software, where you install everything yourself and learn how to configure it (since all of the software is free), do all the networking yourself. This is what actually learning the cloud involves and translating those skills to AWS is very easy.

Perhaps the blame is on the universities and/or tutorials that push the students towards creating free AWS accounts, but regardless, there should be some incentive to takes one profession seriously so you don't end up making this mistake in a company that will not have protections in place that people want for free tier. And money is as good as incentive as any. Id rather people be smarter rather than treat ignorance as a standard and put blame on others for not expecting people to be ignorant.

You are making the assumption that most people are starting these AWS accounts to "learn cloud" when that isn't the case. AWS advertises a free tier for many functions that have nothing to do with "learning cloud".

For example, in the article, the poor student was using Amazon SageMaker to likely build and train models. Blaming the student and saying "why didn't he just buy his own GPU" when Amazon offered 50 hours of free training comes across of out of touch. If the student had the money for a decent desktop he wouldn't be crying over $200.

Amazon could fix this problem easily by introducing a billing cap. While others say it's malevolence as to why Amazon doesn't build one, I think it's more likely that even Amazon couldn't even add this feature to their sprawling billing infrastructure even if they wanted dto.

Can confirm. Default ElastiCache clustering option chooses an extremely high compute node, I ended up accidentally spending $1300 in a month just for testing some Redis clustering script. The minimum option ends up only costing a few dozen dollars a month. The billing alert did not even trigger until the very end of the month, so I got a $1300 surprise. I complained about it to AWS, mentioning how misleading their shit was, and they ended up refunding me $800. Still -- a $500 mistake anyone could make at 11pm. They also made it extra clear in their response that they are not legally obligated to refund me, and that they were doing it out of the kindness of their hearts.

AWS console is some next level outdated shit that needs to be improved. GCP's console is way cleaner IMO. I really hope AWS fixed this after this happened to me, but somehow I doubt they did.

Damn, I can't imagine how you felt... Only the unique experiences in this thread are enough to know how f*cked up AWS billing is.

Felt like a complete idiot, I guess I'm not smart enough to use AWS's UI in a safe way. When we start trusting tools, we inevitably become complacent. Some user flows are designed to be gotcha's. A little gotcha machine that generates |cost - refund| revenue. They gotcha, and there's nothing you can do about it but beg them for a (usually partial) refund. The refund is the thing that makes you feel better about it.

¯\_(ツ)_/¯

Thats exactly the impression I got when I was reading the AWS docs. I am a student too and was evaluating AWS for a hobby project.

Reading the docs I got the feeling that the product is designed for big cooperations only. But not for someone who wants real price control. The free account just made no sense to me.

I ended up renting a cloud server ~3€ a month. (Not AWS) Looking at this thread this was the best decision I could make.

> Corey Quinn, your first and last stop for any question that touches AWS billing, has called for an updated free tier that treats “personal learning” AWS accounts differently from “new corporate” accounts, and sets hard billing limits that you can’t exceed.

This would be good.

I don't normally do "cloud" stuff. It's just not my skill set. But I have looked at it on occasion and one thing that turns me off is my inability to know if I'm going to fuck myself with a large bill from some of these services.

+1 Same boat.

I don't know how to work with AWS. I don't want to pay $$$ for a "course" for something that I can explore with a "free" tier.

I logged onto AWS. Looked around, figured I had no idea what I would be charged and what the "free" tier meant I could do exactly.

So I logged out and haven't been back.

Yeah this is a huge problem. I suspect this is significant for people to choose inferior non-hosted alternatives just for peace of mind. Hard limits would probably create a large influx of users.

Even for something as simple as S3, I'm hesitant to use the real service during development because it's so easy to stack up charges. For one of my accounts, the usual bill is <$10 but last month it was $27 because there was some network flakiness that resulted in excessive transfer.

Yes,they should totally have a high water mark "trigger" so you could set a shutoff for $100 or whatever.

I had a class last semester where we were required to use AWS. Our professor tried his best to teach everyone how to be responsible with their tool use and how to predict their costs (As well as how to set up a free education account and get AWS credits). I was amazed when final presentations rolled around my group had a total cost of 4.30 dollars after 5 months of uptime and use, while other groups had costs ranging from 30 to 150 dollars! I use AWS for my job so I guess I just never really went through those growing pains, but no system should be that easy to rack up costs unknowingly.

I had an AWS account that charged me like $6 per month for a year after I thought I had turned everything off. I finally went on a hunt for what was causing it, and had to escalate to support to find it. This was 11/12 my fault I admit, I should have been on it after the first month.

More recently, I was looking for a cloud GPU provider, and tried a different provider that I won't name. I tried out a ~$3 per hour instance, and shut it down after maybe 20 min. A few hours later I (thankfully) started getting billing alerts that my bill would be $1500 ish at the end of the months if my usage kept up. I couldn't find anything still running, could not get anything from support (they had some kind of chat support that told me to open a ticket), first opened a ticket then after and then after ab hour shut down my account as a last ditch effort. At which point they promptly emailed me an invoice for the $16 I had incurred during the time before I cancelled. The help desk relied to my ticket like a week later, asking for more information.

Needless to say, I won't ever use that company again, but it could have been a lot worse, even more so if I was a student or someone who blew their whole experimentation budget on whatever mistake I made.

> [Vague description of provider now removed]

Hinting at the name like this seems unfair to any other companies that might reasonably be interpreted as "[Vague description of provider now removed]"... especially since I can't decide if you're sarcastic about [piece of description] part.

At the very least [two names] are big names that you could plausibly be referring too... and I could see only one name coming into many peoples minds and then them assuming you're referring to that provider.

I edited my comment if you want to edit yours.

Edited :)

Funny story: I keep receiving a $0.01 monthly invoice from AWS for a very old account (opened 5+ years ago) I have lost access to. I have no idea why (the invoice doesn't list the services being used) and the associated credit card has expired a long time ago. They requested notarized documents to prove my identity. Obviously I'd rather settle my $6 (+ interest) debt 50 years from now than give hundreds to a notary today!

It doesn’t cost hundreds of dollars to notarize something. https://www.nationalnotary.org/knowledge-center/about-notari...

Edit: as the other comment mentions, most banks where you have an account offer this for free. My local UPS store where I have a mailbox also offers the service to me for free.

GP is probably not in Germany, but just an example of why I’d be uninterested in getting something notarized here unless my continued freedom and/or good fortune depended upon it:

American notary: someone who doesn’t have a history of fraud, sat through a workshop and passed a test.

German Notar: a very specialized lawyer, bills like a very specialized lawyer.

Theoretically, you can get stuff notarized at US embassies and consulates, for $50 and an appointment, must bring your own witness(es). That of course has not been an option much of the past year.

I'm not from the US.

In the USA, in Marin County, I was charged $25 per document. The guy doing it was adding a finger print, which is not a bad idea if the originator of the document is present.

(Finger print is not legally required though.)

Any bank at which you are a customer will notarize documents for you for free.

I'm not from the US. This is not available in any of my banks.

If notarization isn't common in the country where you are, you should probably escalate through AWS support in that country. They probably have an alternate procedure. If you're talking to US-based support, they are used to thinking this is a trivial request. Getting something notarized is something that can be handled in 5 minutes at places as common as convenience stores. Most people have coworkers that are notaries and can do it without even getting up from their desk.

If that's not the case in your country, ask them for a different procedure. It's not something they intend to be onerous.

I'm not interested in recovering that account anyway, I'd spend more money (in terms of time spent dealing with them) on trying to get it back than what I'd owe in a few decades.

In Germany you can also get documents notarized in (Catholic) parish offices. I think they do it for free.

That said, this sounds unreasonable and if you're not in the US this may violate consumer protection regulations if they didn't require the same level of verification to sign the contract in the first place.

It is unreasonable but it's just not worth engaging with them even to exchange a few emails, financially speaking :)

I'm not from the US.

Then you have not provided enough information to validate your claim. For example, the EU imposes price controls as well, while in Japan it can be expensive.

However, there's a much simpler route: Amazon is a US country, so you can simply use notorize.com for $25.

My claim is that it's ridiculous to send $0.01 monthly invoices for an inactive account, not sure what you mean by "validate", maybe it isn't that funny of a story? :) But it doesn't make sense for me to spend even $1 on anything regarding this matter. I'm ok with not having access to that account anyway. If anything they should pay me to go through the hassle.

Oracle - of all businesses - got this right. I know, I'm in shock too.

In Oracle Cloud Infrastructure, after your credit is exhausted, you have to explicitly opt in to billing or else they stop all paid services for you.

Moreover, you can choose to keep billing disabled and use their free services without fear of unknown costs.

AWS have just decided to run with a policy of offering refunds when people make mistakes. Unfortunately, some people are ignorant of this, or too timid to ask for their money back.

Wow this is amazing! Never thought Oracle would have such a nice free tier.

> 2 Compute virtual machines with 1/8 OCPU and 1 GB memory each.

> 2 Block Volumes Storage, 100 GB total.

> 10 GB Object Storage.

> 10 GB Archive Storage.

> Resource Manager: managed Terraform.

>Never thought Oracle would have such a nice free tier.

Attracted a fair bit of attention in some corners of the internet for this reason. Then they canned a bunch of accounts using "always free" stuff for myself and others. Do not trust.

> Unfortunately, some people are ignorant of this, or too timid to ask for their money back.

I guess that turns out to be good for AWS’s bottom line.

Might just be an urban legend but doesnt Amazon keep track of people who do too many refunds and cancel services with them? I've heard of and met people who dont do minor returns to Amazon because they were afraid it would put their Prime account on some kind of "naughty list"

I thought I was just to incompetent to figure out how to stop an instance…so I just gave up while watching my creditcard getting billed for about €20 each month.. Last month I went full on and spent a few hours removing everything on every screen I could find. After many hours of failing to remove an instance because you have to stop a thingie from auto starting on another screen first…and another thingie from running there. I finally managed to remove all the instances…

Then two months later, I still get a bill for a few €…

I give up.

I Hate Amazon

You can try using this tool: https://github.com/iann0036/former2

This will scan your entire account and list all of your resources - it's actually made for generating CloudFormation templates, but it's very useful for a use-case like yours.

I'll bet you created the environment with Elastic Beanstalk, whose job it is to (among other things) replace instances if they fail.

When you stopped the instance in EC2, EB did its job and created a new instance.

You eventually figured this out and killed the EB env, and the instances stopped reappearing.

But the Elastic IP address assigned to your EB env is still on your account, and it's no longer free of charge, because you don't have a running instance.

So you will be billed about a €/mo until you delete the Elastic IP reservation.

This is ridiculously confusing, and ridiculously common.

If you delete your AWS account you will still be charged if you don’t remove / terminate some of your services for up to 90 days!

https://aws.amazon.com/premiumsupport/knowledge-center/close...

For me its currently over two years, not 90 days

I used the free tier account to play around for a while. I thought I had deleted everything before the end of the year free tier. I was wrong! For 2 months I paid $10 as I could not figure it out what the heck was running.

I had to close my account in order to stop the charges. Today when I hear anyone speak of AWS free tier for a year, first thing I warn them about is to make sure they keep track of what they create so they will know what to delete, otherwise they will keep buying Starbucks every month to AWS.

Is there seriously no way in AWS/Azure/GCP to specify "Here's my budget, shut everything down if I exceed $X"? I don't use those platforms much but was always surprised I couldn't find anything like that right off the bat. I'll build cloud stuff if it makes sense at work, but if I'm footing the bill I'll stick to something that can provide an actual upper limit.

I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports. You can burn through a lot of cloud resources in 3 days.

GCP at least has some provision to get very detailed information about usage (but not cost) that updates in less than an hour. That, to me, is the tool for building something like "shut down our account if usage is too high". It is annoying that you have to code this yourself, but ultimately, it kind of makes sense to me. Cloud providers exist to rent you some hardware (often with "value-add" software); it's the developer and operator's responsibility to account for every machine they request, every byte they send to the network, every query they make to the database, etc. and to have a good reason for that. To some extent, if you don't know where you're sending bytes, or what queries you're making, how do you know if your product is working? How do you know that you're not hacked? Reliability and cost go hand in hand here -- if you're monitoring what you need to assure reliability, costs probably aren't confusingly accumulating out of control.

Are you being sarcastic?

> I think the big problem is that usage collection is a few days out of date, at least for GCP. Autoscaling can react in seconds to increased load, but it takes about 3 days before that shows up on your cost reports.

That does not sound like a good reason, but more like a crappy implementation of usage collection.

I don’t see why a bunch of Google engineers can’t implement real-time billing properly, and see no reason to defend their inability to do their job.

IIRC the excuse is that billing is a separate department and they count all the usage dollars way after you done using it, not in realtime. You would still be able to go over your limit and then who should foot the bill?

Realtime counting would be too difficult to figure out, our brightest minds are busy figuring out engagement metrics.

This is my personal opinion, though I do work at AWS:

It's not that real time counting is difficult, it is that the amount of compute resources and electricity that would be needed to power real time billing at AWS scale would be astronomical. There is a reason why banks and financial institutions generally do batch processing in the off peak hours when electricity is cheaper and there is less demand for the compute resources. Now imagine AWS billing, which is arguably far more difficult in scale and complexity.