I'm assuming this is related: I can't do any development using Visual Studio 2019 at the moment because it calls home to verify the license and it is currently unable to log me in to to verify my license. So VS is basically bricked for me until this gets fixed :-(
That is awful and why I have always been very sceptical of software moving to the cloud.
Just yesterday I had a similar issue with Resharper, where our local licence server was down for the day. Luckily I was able to enable a 30-day trial. (and I guess Resharper isn't that mandatory to get work done)
It used to be you bought a key to unlock it permanently, but nowadays it ties your license to your Microsoft account. There’s a link to “unlock with a product key“, but I have no idea how you get one of those. I hate renting software rather than owning it, but also VS2019 is where I’m most productive so I put up with it.
This isn’t even the first time this has happened. I do most of my development on my desktop computer, sometimes I go long stretches without using my laptop. More than once I opened up my laptop on a flight to try and get some coding work done, but surprise, license is expired and I don’t have any connectivity to renew it. Absolutely ridiculous.
You used to be able to get keys for all windows software through an MSDN account. No idea if that's still the case, would have been around 2017 last time I had one.
Time travel (reverse) debugging is a killer feature and only available for 2019 onwards. Prior to that you have to use their "IntelliTrace" stuff, which isn't nearly as good.
I have always been very sceptical of software moving to the cloud.
Me too. After a disastrous episode with DropBox while travelling in foreign countries nearly a decade ago, I instituted my own File Server that's accessible via the Internet. All of my data is 'safe at home' with daily backups.
Its less of a hassle to use something like nginx reverse proxy docker container in front of your web services, as it comes preconfigured with some best practices regarding TLS. (TLS1.2+ only by default) https://hub.docker.com/r/jwilder/nginx-proxy
If you use docker, be sure to google around how to manage firewall with containers (must use DOCKER-USER chain instead of INPUT)
Ofcourse, you will get a bit involved in setup step if you choose to tie nginx proxy, letsenccrypt and nextcloud containers. But search engine is your friend.
Not sure if it's a "best practice", but you can get a fairly Dropbox'ish experience by running a nextcloud/owncloud instance. It even comes in docker containers, so setup is quite simple.
Either sftp/ssh with key-only logins (ie. no passwords), or Syncthing (depending on the volume of data, rate of changes, speed and costs of your network transits)
I now keep my primary 'work' and 'private' collections (10-30g each) and a large shared set (350g) synced across desktop & laptop, and also 3 off-site archive servers, using Syncthing. This includes a literal satellite connection for one site.
It was probably my fault in configuration, however the idea was to have a DropBox whereby I could put in files while travelling with my laptop and those files would be taken up with my home server.
Except that it worked in reverse. My laptop was 'emptier' than the home server, so the home server's files were deleted to match the laptop. Hundreds of megabytes of files were destroyed before I managed to stop it.
On top of that, I only had a 2 gig per month quota for internet access while travelling, and all the DropBox activity soon exhausted that in days.
On later trips my own home server coped very well, with less overall unwanted activity.
Yes? I have never heard the concept of "bricking your IDE" before. How badly could you mess up and not be able to reset it? If your IDE can break dramatically without an obvious method to revert to a good state, consider a different IDE.
Welcome to Visual Studio, the home of updates that take hours to install and sometimes end up bricking your IDE.
To be fair, that hasn't happened in a pretty long time for me - but in the earlier days it was commonplace. It used to be bleeding edge features that would do it, other times it was just because they took so long, that interrupting the process of upgrading or installing caused a mess. Or installing the newest version alongside the older version, etc.
Visual Studio is a massive and complex creature (much like other "big IDEs", such as Eclipse) - although it has become much more streamlined in more recent releases - considering how complex it is.
EDIT: To be clear, problems often involved corruption of registry values, library/DLL hell, small database or configuration files and caches, etc. that happened with Visual Studio in the scenarios above.
I've been using it since Visual Studio .NET 2003.. so we as developers didn't all have virtual machines, image-based backups or things like Docker/Vagrant back in late 2003/early 2004.
Yes, since the inception of these kinds of tools, recovery from these corruptions became easier by restoring from an image. Still, in the early days, you could burn a lot of time doing anything of these things!
Yeah I have been using VS for nearly a decade now but I've never 'bricked' it.
Sure, sometimes installing new versions alongside old versions could cause issues. But that's a managed process that one team member would do before the whole team migrated and any issues would be worked out during that stage. (one of our old upgrade steps was to install 2015 before 2013 because doing it the other way around would break something in 2013; 2013 still being needed for some projects).
I have never been in the situation where the entire company (or even entire team) was locked out of their IDE.
Just to back this up, I've also bricked my VS installation multiple times when I worked in that environment. However, some of that was self inflicted when I was trying to figure out how to restore older sub versions of vs 2019 before they started publishing the manifests publicly.
Also, while this isn't bricking the IDE, you can definitely ruin solutions & project files in a very hard to debug way if you ever need to manually update those files.
> I don't think I've ever 'bricked' my IDE. How does one go about bricking an IDE?
Visual Studio major version updates (uninstall old plus install new) used to be a good way to do that and often wreak havoc on the overall stability of your system as well. At least, on a system where you didn't want to strip it to fresh OS install as an intermediate step.
The compiler and the .net platform in general are all accessible without a paid license or internet connection, and actually there’s a free edition of visual studio you can use without a paid license as well. But downloading and installing a fresh copy of Visual Studio Community Edition would take a while and I don’t feel like dealing with that hassle right now. I assume this will be fixed by tomorrow.
Most of the elements you'd want on a status page are there, but the tone and structure are really off. I think you nailed it that this reads like an internal communication to a manager, not to a customer.
Microsoft's "official" status page, which they will usually refer you to, can be found inside the admin center. They've been criticized in multiple previous incidents however, for experiencing an outage then telling people they can get full updates in the admin center, when the admin center requires a functioning logon to a service which is down at the time.
The page you've linked is kind of the "our real page is down" page, and after the issue is resolved what we usually see is all that replaced with something like "go check the admin center for outage information".
It looks like an actual status page written by a developer, instead of some Market speak gobbledygook attempt to not say anything for fear of liability or whatever
It’s the “oh crap, there is an unsurmountable obstacle in between the end users and the admin portal. We have to say something now” page. Most of the time, service alerts are supposed to be reported through the Azure or MS365 admin portals.
It looks like it was copied from a text file, since if you view the source the actual formatting is there; but HTML's default whitespace/newline treatment causes it to turn into one big blob of text.
> Current status: We've identified a recent change that appears to be the source of the issue. We're rolling back the change to mitigate impact.
Current status: We've identified that reverting the recent change did not alleviate impact to Microsoft services as expected. We're working to explore additional options for mitigation.
[I actually enjoy these updates but they indeed read more like our internal incidents channel]
The cloud is the ultimate rent-seeker's dream. The world's computing power in the hands of a few enormous companies... who then sell it to you in tiny pieces and charge for every little thing you use on their services. And when something goes wrong, you are powerless to do anything about it.
This is why my new strategy is to avoid purchasing mission-critical components of my infrastructure from gigantic vendors.
For example, right now I could be using Azure SSO, which would have been free with Office 365 at the level we're bought in at, and then all of my other cloud products would be just as inaccessible as Office 365 is. Even when it is working, I could have an edge case issue and never see a feature added or a bug fixed to help me at all. Or, I could go with a 3rd party authentication company, and get a product that their company lives or dies on. My experience is that, if chosen well, you'll get a much better experience even though you have to do some legwork here and there to make things work together.
The sweetspot of cloud is that it is more expensive to operate your own hardware alone, but cheaper than operating your own hardware with paid professionals to manage those hardware.
I just finished convincing old-school management to move to Office365/Azure/Teams after we've been doing everything on-premise, saying we can reduce our maintenance and increase in reliability since we're a small operation. Needless to say this isn't a good look for me.
If it makes you feel any better, I've been that guy on both sides of the coin - both on the cloud and on-prem. Things go down.. even the cloud.
I worked for a company, where I really did have nearly 100% uptime at my local data center in our office. I wasn't a luddite, and the cloud wasn't something I was afraid of or didn't understand. I just, at the time, had an infrastructure in place before the cloud existed, and it worked well enough at a great price point. We were on the same electrical grid as a hospital, our infrastructure handled our scale, and just really didn't have problems for a long span of time.
Still, someone came in and said.. what, you are still doing stuff on-prem and not the cloud!? How legacy! How dated! And so the push to the cloud came next.
We migrated to the cloud (and some colo facilities for some equipment we chose to keep), and on the 3rd day of being freshly migrated, the cloud provider had a major outage and went down for longer than we'd ever had on prem. The next month, the colo went down and their diesel generators failed, and it was down for an entire afternoon.
Oh sure, there was some SLA money returned in that event from the colo...
I'm just saying, I've been on both sides. Tell them the truth - even the cloud has outages, but it is certainly more "convenient" to have hundreds of engineers work on fixing an outage at global scale, and all you have to do is wait for it to start working again - then it is to have to fix it all yourself.
> Tell them the truth - even the cloud has outages, but it is certainly more "convenient" to have hundreds of engineers work on fixing an outage at global scale, and all you have to do is wait for it to start working again - then it is to have to fix it all yourself.
Yep, that’s the argument I’ve made, both to management and even our non-techie customers. When we show them our new SaaS product, they get concerned about cloud outages. Our response is “if AWS goes down, you have bigger things to worry about since other things will be down too.”
The plus side I’ve mentioned that our company’s software developers are freed from doing network IT maintenance and infrastructure and can continue focusing on our actual core products instead.
Why are your software developers doing "network IT maintenance and infrastructure" in the first place? Is it because of "DevOps" maybe?
It's one thing to experience the sysadmin practise getting taken over by developers. The next thing is usually the latter needing to be "freed" from this burden... Seems like a big fad.
The variable here is not just the size & skill of the team but their needs. In most cases a single company or team will need a simpler, smaller-scale deployment, probably running on one or a handful of machines and not requiring the insane complexity that a public cloud provider is running behind the scenes to provide their service.
This eliminates a lot of moving parts and things that can fail. Imagine an Nginx server running on a single machine. There's very little that can go wrong here beyond hardware failure (and certain types of failures can be mitigated with things like RAID), and yet it is probably enough to host most internal websites.
Now compare this to something like Azure App Service which is obviously more complex to be able to support many tenants, load-balance, etc. There is much more that can go wrong with the entire App Service infrastructure (due to its complexity and moving parts) than with a single machine running Nginx, and the complexity will also delay disaster recovery efforts (this outage is now lasting for more than an hour - you can reinstall an entire Linux web server from scratch in half that time).
If they have enough resources. When everyone is busy working on the revenue generating products for the company, taking time away to work on internal network infrastructure immediately becomes a cost center.
Microsoft Teams (The chat/collaboration service) and team foundation server (The source code repository and DevOps platform) are two completely different products that happen to both have the same word in their name. Although I can understand the trauma ;-)
At least the guy with the change request was vindicated for now. That must have felt good - about to be crucified for putting in a change that brought down AAD globally, they roll it back and shit's still broken. :D hahaha wudn't me mofos!
I've gone all in on Microsoft for my company (high Office usage) with O365, Azure and friends and I'm regretting this decision.
OneDrive for Business is almost unusable on Windows 10. The web apps are incredibly slow. Teams is very inconsistent for users between desktop and mobile usage. It's a mess.
I'm planning on switching to Amazon next year. I hope I can find an alternative to the Office situation (maybe offline licenses).
I have found a Dropbox to be more reliable at file sync than OneDrive, especially if you have many, many small files. Then again, the pricing is so far off between the two that I can’t justify Dropbox anymore and just put up with the horrible OneDrive performance. Does anyone have any better alternatives?
Luckily Microsoft still sells offline licenses for Office 2019. There are several features missing that are exclusively tied to the Office 365 subscription, but overall it’s still the same Office productivity suite you would expect. Here’s to hoping they keep doing that in the future and don’t pull an Adobe…
If I were offering a free (as in price) alternative to MS Office, I'd first recommend WPS Office over LibreOffice, even though it's closed-source. It has a much better interface out of the box. But I'd use something to block it from phoning home to its probably Chinese government-beholden servers, like OpenSnitch for Linux* or LuLu for macOS.
If I may, in my experience Softmaker is not bad at all (still closed source and Commercial, but affordable), LibreOffice simply does not fit, for very light use LibreOffice is fine, but still has - here and there - too many "quirks".
Ahh, thank you. I simply dabble in PS on the side and write util-like scripts as a brain stretch. This is the first time I've ever had someone look at and 'clean' anything... much appreciated.
> We're not observing an increase in successful connections after rolling back a recent change. We're working to evaluate additional mitigation solutions while we investigate the root cause.
Yep, I can imagine the feeling of doom when the rollback didn't work. Push back from the desk, head droops down, hand on forehead. Then I whisper, "Fuck".
O365 (aka Microsoft 365 now) has easily been the most unreliable SaaS product I've been forced to use over the past 8 1/2 years, it's been a real disaster story for several of the clients I used to have and for several of them has resulted in loss of profits and general distrust in Cloud/SaaS software.
A few times during outages I managed to get hold of someone that worked at Microsoft who stated something along the lines that they only state that a service is degraded or has an outage if enough customers complain - or if the websites they have to sell / demonstrate products to potential customers are unavailable.
"Current status: Rolling back the previously described change did not resolve the incident as expected. We're evaluating additional options to remediate the problem."
We (like many other orgs, I expect) have gone all-in on AAD, so this has pretty much taken down everything company wide with the exception of the factories.
Good thing the plant floor isn't tied in.
I know many posts in this thread have been jokes, but I do hope this outage helps MS reconsider their offline iterations of Office. I use Office on a near constant basis (writing and building lecture PowerPoints). The subscription model isn't feasible and would ultimately be more expensive in the long term. When Office 2019 was released I quickly picked it up out of worry that MS would shift their academic licensing programs to 365, which they ultimately did.
There was a big one in 2018 involving authentication. A handful of datacenters outside of Houston couldn't get enough power and the shutdown procedures prioritized saving data. Some critical Azure system service was only located in that geographic zone and the data recovery priority caused that low level service to fail its migration. It caused a huge authentication failure across all Azure services globally. MS got blasted about it at Ignite pretty soon after.
Azure DevOps' Git is also down (a git pull hangs forever) which is a bit surprising. I wouldn't have expected SSH key authentication to depend on Azure Active Directory.
Microsoft Teams also appears to be down so it's not just authentication that's affected as I was already logged in and it still doesn't work, just hangs trying to load forever.
Edit: appears to be resolved at 1:07 am London time.
I noticed some strange behavior around 3-4PM central. I was searching my inbox and saw an error in Outlook "we're unable to search the server" or something like that. I have never seen a message like that from Outlook before.
My kids' school district is doing all online over MS Teams; things started crashing and burning around 3PM PST (no "Dad Support" can fix MS's servers though).
is this a national risk? - think other countries taking out these massive, saas offerings. must be easier than simultaneously targeting thousands of disparately configured, onprem implementations
This is the first time since 2015 that I remember an outage of this magnitude (scope + impact) with the platform. Not saying my memory is authoritative, but.. I certainly wouldn't call this regular.
Screenshot: https://rog.gy/ss/3b14f7a2ac.png