The options aren't just give $10B to AWS, Oracle, IBM or Microsoft. The JEDI should have been a multi-award IDIQ (indefinite delivery indefinite quantity) contract. With a multi-award IDIQ the each task order issued under the IDIQ is competed for among the multiple vendors the IDIQ has been awarded to. It is actually fairly unusual that this type of IT services contract was not done as a multi-award IDIQ.
Having to compete for each task order is thought to reduce costs to the government. Of course having to compete each task brings its own additional costs which don't necessarily make the cost saving proposition true. The big benefit I see to this contract is that it helps prevent vendor lock-in. I don't worry about the next 10 years and $10B, but what happens after 10 years when the DoD is now locked in to proprietary AWS cloud features which make migrating to a different vendor technically and fiscally near impossible.
You should not do infrastructure as IDIQ, this is an infrastructure contract. The idea that all contracts issued by IT are for "IT Services" is part of the reason digital systems at the DOD and other government agencies are such a nightmare. HAving a single system to build off of is the best way to do it, and not having that system be a customer solution for just the government is way better then not.
Lock in is the only concern, but from my experience doing an IDIQ or multi-vendor won't solve the lock in problem. The best thing to solve that is to make sure your using bland normal things not customer government thigs.
You know, I've been in some data centers and top 50 HPC clusters ... I don't understand why the US government doesn't run it's own cloud systems, except for the appropriations. There are enough computer scientists in the larger org to field a major distributed systems engineering group.
Now, think of the political climate: there’ve been decades trying to “trim waste” by steering money to the private sector, where efficiency is taken as a matter of faith. First you’d need to hire the technical leadership to properly plan something like that and then hire all of the people who’d build it, not to mention procuring all of the hardware & data centers, etc. You’re looking at years before you have a positive return on any of that – and all the while, every big contractor is going to have their lobbyists talking about how it’s a waste of money compared to going with them.
NASA tried this a decade ago with Nebula, which was part of the early OpenStack history. I don’t know exactly what happened with that but the people I know working on NASA projects are all using AWS.
I'm glad you mentioned the caveats with IDIQ contracts. I would say that the IDIQ process rarely, if ever, results in savings. (In fact, if you have some examples where it did I know some people who would be interested in studying those cases.)
The history of IDIQ processes is likely a large part of the reason people were anxious to do something different in this instance. Now that it's all politics, it'll probably cost more and function less when all is said and done.
Some aspects of the military, I don't really miss.
I’ve done IDIQ contracts in state government. You can absolutely save money, but you need to force a limited list of SKUs and renegotiate pricing at different volume-based high water marks.
Winner take all is also important, because it eliminates the room for sales bullshit. The magic is figuring out how to break up the lots of services so that it makes sense. You don’t want to give any one of AWS, Google and Microsoft everything, so you want to design lots that keep it competitive. This also ensures that any one vendor doesn’t subsidize strategic SKUs by discounting something that isn’t meaningful.
Everything fed is more complex, but it’s all doable.
What’s your theory on the justification DoD is giving to go single-award IDIQ?
I doubt their actual justification will ever be public, but I suspect their motivation is that they require a common IaaS platform such that they can have a unified toolset and a single, comprehensive approach to infrastructure security.
Keep in mind also that fewer proprietary features at AWS are available when considering standardizing across all classification types. “East/west” and GovCloud don’t have feature parity.
Especially since that 10B becomes 20B when in 5 years, Oracle tells you that you that your initial implementations are being EOLed and you need to start over.
I've been torn on this contract from a professional experience POV. In the IC we got AWS after they beat out IBM and the IBM protest. I'm an AWS fan so I was excited and it worked well for me when I was a contractor (I'm no longer in the space).
But having the DOD on AWS along with the IC concerns me from a lack of diversity standpoint. I've always said that Microsoft and Google should have teamed up a bit to make a competitive run. But I don't think anyone was ever interested in sharing the contract to begin with.
Azure has a large presence in the government and they have a fairly robust offering that multiple agencies use. So its not as big a mono culture as you might think. I know this one contract makes lots of noise because its the DoD and people think the military is this huge organism, but the Department of Veterans Affairs was on tract to spend 1B on cloud per year and they use Azure ( as well as AWS )
It blows my mind a department dedicated to Veterans Affairs can spend 1 billion on cloud computing in a year, do you have a source or any more information about what they’re spending that on?
They're probably the largest hospital provider in the country. Looking at their wiki it says there's 152 hospitals and 1400 outpatient facilities. Plus all the other VA responsibilities they fulfill.
It's a big organization and they have a lot of 'customers'
Because you’d rather have wasteful spending than moderately productive spending...? Seems like you want the government to waste money just so you can say “I told you so”.
Like it or not, defense spending is a huge boost to the middle class stem job market, and taking it out would have consequences wider ranging than you think. With “white collar welfare” I’d argue the gov gets a better return on its investment than most other options, while arguably making it harder for developing nations to become a threat via brain drain because the stem market is strong here.
Yes, precisely—wasteful defense spending accomplishes the goal of supporting the middle class with jobs while failing at the goal of killing people. It's not an "I told you so" thing. I'm genuinely fine with massive taxpayer-supported boondoggles that create jobs in the long term. My objection to defense spending is that it kills people, not that it takes money from my paycheck to create more paychecks.
As you may remember from the Project Maven controversy, there are a lot of people on HN and in the tech community who are fundamentally opposed to the US military being more effective at carrying out its missions.
Oh Oracle - first your sales people came every year and jacked up the licenses, and we fled.
Then you came up with plans to monetize Java, and we fled.
And then this.
Just die already, OK - your old business model is dead and your attempts to make a new one would even embarrass Elsevier
I seriously wish Google had bought Sun. Their cultures are so much more compatible, and while the Sun's hardware business didn't really mesh with Google's, it would have saved them some headaches with Oracle, and with the direction custom hardware is headed, might have become a real asset.
> your old business model is dead and your attempts to make a new one would even embarrass Elsevier
Sounds like wishcasting. Oracle is still be raking in money. The death of Oracle has long been a fan favorite on HN, but it's nowhere near being dead and their business model while perhaps antiquated by HN SaaS standards is alive and well and still seems to be working though they're diversifying into a number of new areas to deal with possible long term decline of Oracle DB.
I don't know how they do it, but I've seen some really diehard anti Oracle companies adopt Oracle. One place had a really large mainframe installation and DB2 licensed all over. They moved to Oracle. I was shocked. They play some kind of massive long game, no one else is at their level when it comes to sales.
Larry Ellison is exactly the kind of guy Trump loves. He'll get one flattering phone call and the contract is theirs. Then $10B will turn into 100 before they hit the first release.
Lots of drama to this story, between the apparent sabotage from Oracle to the politics of the president not favoring AWS just for spite, to the Pentagon's acronym for its cloud move: JEDI.
The article states that Microsoft does not yet have the security clearances for JEDI yet, but is working on it, but nowhere in the piece does it mention Google's cloud services. Are they in the running as well? Are there any other giant enterprise cloud platform businesses that I'm unaware of? Dropbox uses AWS.
The other interesting thing about this, circling back to Oracle, is the changing of the technological guard here. One can imagine all sorts of government databases running on IBM mainframes, using Oracle software. But now, the shift to enterprise cloud has arrived. I guess this must be happening across the entire industry.
Google Cloud saw itself out the door in the JEDI competition quite some time ago. Competitively, this decision hurt their business in a number of ways, and led to a huge fallout (leadership churn).
Wasn't it because they hit the scale where managing their own data store, renting colo space, etc. was cheaper than AWS? I've heard first-hand from some large players that self-hosting is cheaper. One is still-hosted, but one is on the cloud as a strategic bet.
Cutting out the middleman and their profit margins is almost always going to be cheaper if you have the requisite scale requirements. Depending on what your workloads are (particularly non-bandwidth intensive), that scale can be pretty damn large for compute infra nowadays.
IMO, I think you're undermining the multiple years of programming/sys-admin experience it would require to self-host + scale on your own. It's much more expensive when experience is minimized, as a function of time.
I can't say authoritatively, but the times I've talked with NSA admins (who of course didn't say they worked for NSA, but DoD) they used a lot of IBM stuff, not the least of which is because IBM would handle support requests on Christmas Eve.
Well, that's just a clever PR lipservice move. They couldn't win the contract because they didn't have the ability to, so rather than failing embarrassingly, they blamed their exit on their brand-new AI principles to make themselves look good. This was the consensus on HN at the time.
I know we tend to think of cloud-agnostic architectures as over-engineering, YAGNI, etc., but for a massive, massively expensive, mission critical government cloud perhaps it’s a good idea?
Apart from fail-over/redundancy benefits, it would avoid lock-in and drastically improve negotiating leverage that could force ongoing competitiveness and accountability.
The engineering costs are significant but still a drop in the bucket compared to the size of these mega-contracts.
A contract the size of JEDI can (and probably is/will) build in that kind of independence as a requirement. Which isn't to say "Amazon must build and maintain an OpenStack implementation for DoD", but rather, that Amazon must turn over all APIs and source code used to build the implementation. Obviously, source code and APIs isn't enough to keep everything running - there still needs to be operational expertise within organizational memory - but the DoD would be empowered to start to take over operations and plan a migration to some other company maintaining the cloud infrastructure, if it were really untenable to renew the contract in a decade.
Put it this way: there's an old aphorism about how owing the bank $1 million is your personal problem, owning the bank $1 billion is your bank's problem, and owing the bank $1 trillion is your country's problem. There would be a parallel for cloud infrastructure: if you have a $1 million cloud bill then you have a (vendor lock-in) problem; if you have a $500 million cloud bill then your vendor has a problem (because you represent a huge chunk of business that can evaporate if you move to on-prem); if you have a $1 billion cloud bill then the wider market would have a problem (because of the reverberating shock effects if your migration to cheaper infrastructure is unsuccessful).
It's the textbook definition of something which is too big to fail.
"the DoD would be empowered to start to take over operations and plan a migration to some other company maintaining the cloud infrastructure"
That's still a massive project though that I presume would only be undertaken in extremely dire circumstances, so Amazon only has to do the bare minimum to make sure things don't reach that point.
I'm thinking more along the lines of switching off the primary cloud to a "follower" cloud in a few clicks. I realize that's not at all easy, but I think it should certainly be possible with these kinds of budgets to build in cloud-agnosticism from the ground up. It's hard but it's not that hard--it's just the facade pattern, which is quite commonly applied in other cases like OSes/databases that are arguably just as complex.
I think "extremely limited" is hyperbole. The major clouds can all provide every service that 95% of applications need. If you're building something with unique needs that only one cloud can handle, then go ahead and use the advanced feature. A little bit of lock-in here and there for edge cases seems preferable to having your entire multi-billion dollar system deeply coupled to a single company's cloud.
Sort of. AWS Lamba (server functions), dynamodb (hosted nosql) and aurora (hosted sql) are all pretty generic services, have significant auto-scaling benefits to using them, and have competitors on competing platforms.
At the same time they are sufficiently differentiated that migration plans would be non-trivial
Obviously there are lots of ways you can vitiate your ability to do this by being pathologically dysfunctional, all I'm saying is that the proposition isn't a priori unreasonable.
Your objection seems flawed to me in that it ignores the fact that goons can screw up just as badly on AWS as they can in a data centre.
My first thought is spaffing huge piles of money up the wall by using expensive stuff like DynamoDB really inappropriately, but I think in general AWS is complicated in a way paying someone to plug in colo servers isn't and thus offers more scope to waste money. For random anecdata purposes, I saved $20k /month turning off unused stuff at my ~$50k /month most recent gig. Now scale that up to $1B /year
I think 3 way redundancy would be about right for a massive, mission critical government cloud.
Apart from fail-over/redundancy benefits, it would avoid lock-in and drastically improve negotiating leverage that could force ongoing competitivenes and accountability.
The effort could be open sourced, which would have industry-wide economic benefits.
The department of defense, and any contractors in the running for that "cloud-agnostic architecture" do not have the capability to provide a service the would be 50% as good as what they would get out of the box with AWS. (or azure, or GCP).
Half the shit the gov buys and mandates people use nobody doing work actually wants to use. Picking Oracle or some two week old fly by night hub zone contractor to handle this award would be typical. The unusual thing would be for it to actually go to a provider with services that are useful and people want to use. - speaking as someone who has done a significant amount of .gov work.
The government already contracted General Dynamics to build their own cloud called MilCloud. It had a lot of problems so naturally they are developing MilCloud2.0
I guess I don't understand the need for this contract in the first place.
Is it normal in the private sector for companies to spend $1B/yr on cloud services? I feel like at that price point you're better off building out your own infra for most stuff, and using the cloud only for the remaining $10M - $100M fraction where you really do need dynamic scaling.
I guess owning the Washington Post (Bezos paper), and its overwhelmingly anti Trump stance over the last 3 years, had its price. This is about Bezos IMHO, nothing else.
WaPo has been pretty fair if not overly gentle. Trump has mostly been flipping out over pretty bland and factual reporting that happens to be embarrassing for him because he keeps doing embarrassing things. And it's pretty grossly unethical to use your office to punish the media.
I'm no fan of trump, but in what way have the media been punished by the office?
I get that there have been incendiary tweets indirectly resulting in private citizens being assholes who have made death threats against the media. Clearly trump is an asshole for doing those things. Have there been any official punishments by the authorities that I'm not aware of?
Pulling press pases, just about ending press conferences, Calling everything fake news. Current admin hates the media unless it reads from it's talking points.
Calling something fake news is not a punishment (you could construe almost any action done by a president as a punishment for someone under any criteria sufficiently vague as to admit that), and pulling a press pass is hardly a punsihment given how farcical this governments press conferences are. Have any journalists been jailed, murdered, fined, or had a business licence revoked by the state? These things happen, and also in 'civilised' nations like EU member states. If trump has done this I would certainly appreciate knowing so as to criticize the administration for it.
It is when the highest and supposedly most-respected seat in a country undermines the entire validity of the fourth estate in a time when there's no viable replacement. It's a personal, poisonous and Pyrrhic sort of punishment, but it's punishment nonetheless.
The whole bloody point of the bureaucracy and institutions is to not care about the day to day politics and get the job done. That is why they have their protection and civil service exams.
The fact it is being treated as relevant is deeply wrong and corrupt.
The thing is: being critical of the government is the literal job of journalists - everything other is propaganda. The press is supposed to be part of "checks and balances" - and the way Trump regularly attacks all press that doesn't lick his boots (=everyone except Fox) is worrying.
And on the other side, the government should be impartial and only look at the financial and technological sides when procuring stuff or enforcing regulations. Trump's opposition to everything Bezos is obviously everything but neutral.
>>>The thing is: being critical of the government is the literal job of journalists
Except the bulk of them aren't even competent journalists. They hardly do any fact-checking or ground-level reporting. They are closer to socialist agitators than journalists. Take the Baltimore rat Tweet situation as a recent example (The "Covington kids" are another good example.) One America News interviewed Baltimore residents [1] and got a completely different vibe from what the MSM has been pushing. Meanwhile CNN's anchor with his forced fake tears about "Trump's racism" is getting savaged by black YTers [2][3][4]
>>and the way Trump regularly attacks all press
"All press" != "CNN/NYT/WaPo/HuffPo/MSNBC". Has Trump attacked Tim Pool? No, he invited Tim Pool to his White House Social Media event, despite Tim constantly calling Trump a crass person of bad character. Tim Pool gets more daily YT views that CNN gets TV viewers, BTW.
>>>And on the other side, the government should be impartial and only look at the financial and technological sides when procuring stuff or enforcing regulations.
Agreed. Or they should pay technical specialists what they're worth and develop and maintain internal government solutions. For some reason that's a bridge too far for us.
I genuinely wonder sometimes if we are on a slide towards a banana republic, and it makes me genuinely concerned that many people (40-46% of them) just wouldn't care if we did.
They would happily live in a banana republic just to spite the "libruls". Some people simply are not worthy of the privileges of living in a democracy, it seems.
This headline seems... misleading. There's only one high places foe mentioned, and that's Trump. Every other high level official mentioned seems to be doing Trump's bidding on this.
Also, how is having someone high up intervening in a business process a big violation of free market principles? I don't recall the article mentioning this either. As I think about it, this just reeks of crony capitalism. Didn't we just (2008 or so) decide that was a bad way to do an economy?
Isn't the NYT slant generally in favor of laissez faire free markets? Why the switch here?
You're right, but unfortunately the idea of a President individually attempting to punish individuals and businesses that criticize him is considered normal now.
What is it that the government is buying and how can the government ever trust (for this sort of thing) a private company. And surely you can set up faster (performance per buck) more secure data centres for 10 billion bucks?
I just don’t get it, but I guess someone will say government can’t do anything and that’s enough.
The worst thing about this in my opinion is that as much as I personally dislike Trump, the WaPo has lowered the quality of its journalism tremendously over the past decade and so unfortunately Trump’s critiques of the WaPo are not lacking in substance.
Bezos should have stepped in to make certain that the WaPo improve the quality of its journalism even if the easiest path to profits is just to provide entertaining stories to the #resist partisans.
There is plenty to write serious investigative journalism about that will have the consequence of weakening Trump politically (if that is the goal) so there was no need to become a tabloid.
Arguably the WaPo’s reporting on Saudi Arabia led to the grudge match that culminated in outing Bezos’ philandering, and the subsequent major decrease in Bezos’ wealth.
KSA is led by an abhorrent group of royals, but the WaPo’s coverage of the disappearance of khashoggi was tabloidesque and immature.
>WaPo’s coverage of the disappearance of khashoggi was tabloidesque and immature.
One of their writers was murdered in a foriegn embassy on foreign soil by the royal family of SA. Tortured and murdered, and dismemembered.
That story deserves quite a bit of attention. I'd also like to hear your recommendation on high quailty journalism since WaPo does not meet your standard.
There hasn’t been any real information about what happened to the journalist, or about any other entanglements he may have been involved in. There is a good chance KSA wanted him dead, but most in the US do not take seriously the question of whether the US stance toward Saudi is flawed. The US just sold KSA massive amounts of weaponry.
The Intercept has solid journalism. The New Yorker does also, some of the time.
Journalistic standards have been abandoned by the major news orgs in favor of in group targeted stories that are activism oriented (in the left vs right partisan fray) and not truth seeking.
It has been clear for a long time that the WaPo and NYT are playing partisan politics and are cashing in their previously earned journalistic credibility to do so.
Both papers wasted the first few years of Trump’s presidency with front page stores about his manners and buffoonery, with no serious investigative work until the story about Trump’s inheritance earlier this year.
It’s as if the reporters think their job is to “discover” the blindingly obvious fact that Trump is crass again and again while ignoring the war crimes and bipartisan evil that has been progressing nicely since Trump took office.
> There hasn’t been any real information about what happened to the journalist
You and I live in different realities. There is conclusive evidence as to what happened if you bothered to look. I can only assume you are not speaking in good faith.
Re: the rest of your post, AGAIN if you bother to look, there is quite a bit of investigative journalism of substance regarding Trump's misdeeds. Seriously decades of it. I don't know what to say except to try using Google, the information is out there.
> In Pentagon Contract Fight, Amazon Has Foes in High Places
> Bezos should have stepped in to make certain that the WaPo improve the quality of its journalism
The relevance is that this waffling over Oracle and AWS is born of an interview in WaPo? It has nothing to do with Amazon. Especially when you're commenting on the quality of WaPo journalism. The story exists without WaPo, but not without Amazon.
WaPo has been weaponized against Trump by Bezos, or Bezos failed to stop it.
For some reason doing solid investigative journalism is not a priority and instead rallying partisans around talking points and feel good, in-group stories takes precedence.
FWIW I personally loathe Trump and would love to see him removed from power thanks to some solid investigative journalism.
I don’t think that the irony should be lost on you that this generation of service offerings do not guarantee reliability on purpose. The onus is on the developer to make a reliable system out of faulty pieces.
What does more reliability mean when that was expressly not a target to begin with?
