Yeah, that's why Epic Games have been transparent enough to post this incident report: not to provide some explanation to their customers, or some information that the rest of us might be able to learn something from, but so that people on HN can make entirely unfounded accusations about the state of their organisation based on (at best) weakly correlated behaviours and symptoms.
Be reasonable: you know nothing about how Epic Games treats their IT staff or whether or not the team is adequately resourced. I wouldn't say certificate expiry is something that happens particularly often, but I have seen it happen, and it's been simply an oversight rather than an indication of some serious systemic issue.
The fact that a company can't deal with a scheduled-far-in-advance, highly-public-if-failed event does tell you some things about their priorities / how well they do things they need to do.
And in recent years they've been crippling extensions more and more, and even completely dropped support for them from their primary mobile browser for over a year now.
So yes, I think this is one of many signs that they're not paying enough attention to extensions, not a totally isolated "accidents happen" event. Were I an extension author, I'd see that event as reason to be more concerned.
> and even completely dropped support for them from their primary mobile browser for over a year now.
You're misinformed. Many extensions work, they are progressively being re-enabled over time, and on the nightly version they are all available, although whether they actually work or not depends on the state of the underlying APIs. The reason for the whitelist model, is that when they swapped to the new mobile browser engine, the underpinnings of many of the extension APIs had to be reimplemented, and they are not all online or bug-free yet.
The nightly browser, and the ridiculous[1] steps you need to take there to use extensions, are not their primary mobile browser. Installs from Google Play alone are two orders of magnitude apart.
And no, I don't consider a small custom list to be "support". It's a high-value list and a solid sign that they're not wholly abandoned, and I do expect it to come eventually, but it's very much not the same as general availability. General availability did exist before.
Edit: I broadly agree with their breaking of NPAPI stuff, WebExtensions (as a concept, not necessarily the specifics we have now) has a LOT of very real benefits, and does not inherently prevent equal or better capabilities. But it too is still a loss in control, as it stands today.
AWS has failed at this numerous times. A company filled with a large number of incredibly smart, incredibly capable engineers who have routinely failed to notice certs were about to expire etc. Amongst organisations with operations teams obsessed with uptime, and an obsession about measuring and monitoring everything.
I sat through so many internal incident reports that boiled down to expiring certificates there.
Since I left, I understand that they've fully automated, and mandated, all certificate generation and rotation, but there have still been cert expiration events, albeit rare.
Cert expiration events happen. It's zero indication of the intelligence or capability of the engineering skills or maturity of a company. It's a thing that just works until it doesn't, with zero warning.
They don't have to be renewed on the very last day - non-zero warning is easily achieved. Renew it a month early, and look at it at some point before that month is up.
That they apparently sometimes fail to do these things.
You can't verify anything internal unless you're internal or it has already failed publicly, so you of course have to draw on patterns seen elsewhere. Critical-process failures in one area correlate heavily with failures in others.
Plus, Epic has not exactly shown themselves to be producing consistent quality in anything related to their store, or many internet-connected properties. If they were, this might be more attributable to "accidents happen, it's impossible to prevent them all". It could still be an abnormality, but they're edging further towards "... maybe not though" territory.
---
Edit: lets add a concrete "kinda example, kinda counter-example". Google is a tech company that is pretty good at consistently renewing its many certificates. They recently failed to do so for Google Voice: https://www.bleepingcomputer.com/news/google/recent-google-v...
I think there's a reasonable argument to be made that this reinforces claims that Google Voice is low priority / at higher risk of future issues due to lack of care, i.e. systemic issues, compared to other Google properties. I have no proof, but that doesn't mean it's automatically unreasonable.
Sure, but you can't actually use an example from Google to deduce what's going on at Epic Games.
Don't get me wrong: I'm not saying there aren't problems at Epic Games (most companies have them). What I'm saying is, we're just speculating: how is that helpful? Either to them or to this discussion?
We're either casting vague and hand-wavy aspersions or citing more specific examples where we actually have no idea whether they have any relevance to Epic Games.
It's just noise because, as you've pointed out, we're not internal.
Are you arguing that the internal workings of a company can't be visible at all to outsiders? Or that there's no correlation between the rate of public, easily preventable failures and technical incompetence? Or just that it's not "helpful" somehow to point these things out?
Epic games published this as a PR move. Nothing more, nothing less. Customers got mad because Epic fucked up so they had to say something to make it seem complex and totally reasonable.
“We made a bad bet on certs not being that important, it backfired” doesn’t sound good but it’s the truth.
The same thing happened when Delta got wiped out by a power outage. “We made a bad bad bet on geo redundancy not being important, it backfired” wasn’t good enough for them either so they pontificated just like Epic did here.
It’s obvious that Epic doesn’t take certificates very seriously here. This is cert management 101. No need to read into it much further.
Be reasonable: you know nothing about how Epic Games treats their IT staff or whether or not the team is adequately resourced. I wouldn't say certificate expiry is something that happens particularly often, but I have seen it happen, and it's been simply an oversight rather than an indication of some serious systemic issue.