Yesterday, a “severe weather event, including lightning strikes” caused a “voltage power increase” to the cooling systems that in turn caused a controlled shutdown of at least some of the hardware to prevent data loss. That, in turn, caused users around the world to lose access to Visual Studio Team Services, and to some services of Azure Active Directory and Office 365, affecting users around the world.
This morning, the Azure Status page still shows warning signs on all aspects of the South Central US data center, and reports as of 13:00 UTC (6 am Pacific time):
ENGINEERING STATUS: Engineers have restored access to storage resources for the majority of services, and customers should be seeing signs of recovery. Engineers are continuing to work on any residual storage impact to fully mitigate this issue.
The Visual Studio Team Services blog also reports progress, although for some users, work is still to be done to gain back access:
Update: Wednesday, September 5th 09:50 UTC
Recovery efforts continue.
South Central US region is partially recovered, a subset of customers will be unable to access their accounts or face performance issues. We are working to mitigate those failures.
Extension search powered by VS MarketPlace service will face intermittent failures while loading, we are in the process of upgrading the impacted SQL Azure database to mitigate the issue.
For some users, of course, who have come to rely on services like Visual Studio Team Services, the outage has come as a blow. This tweet reply to one of Alex Simon’s reports yesterday sums it up:
VSTS still not working. My company loves burning hundreds of thousands of dollars an hour in lost dev time.
— RvLeshrac (@RvLeshrac) September 4, 2018
So in general, services are coming back online, some users are still affected, and work continues. We’ll keep you posted, including any postmortems of what happened and why.
