Microsoft said it had stabilised swathes of crashing servers six hours into a sweeping global outage – as network data showed how extensive the impact had been.
Redmond’s engineers had appeared to be struggling to identify the cause of the Microsoft 365 outage six hours after it started– which also affected Microsoft Defender cybersecurity services and remote management toolkit Intune.
In a series of status updates throughout Thursday, Redmond first suggested that elevated load on a portion of infrastructure might be to blame – and began by “redistributing connection traffic across our broader worldwide infrastructure.”
That didn’t work.
It then identified that a “number” (likely a large one, given impact) of servers were failing Microsoft’s automated health checks and becoming unavailable – straining adjacent infrastructure. Arguably unusually for Redmond (which often blames buggy configuration changes for availability incidents), it said several hours into the outage that it had “not identified any recent change correlated with the start of impact.”
The incident appeared to be affecting the admin center for many users globally; leaving customers unable to get detailed service health information there.
Data from Thousand Eyes showed (below) the scale of network failures globally to be significant - with availability starting to tumble around 17:30 GMT/9:30PT.
“Beginning around 17:30 UTC, ThousandEyes observed availability issues affecting Microsoft 365 services globally. Users attempting to access Microsoft Outlook and other Microsoft 365 services encountered connection resets, connection refusals, response timeouts, and HTTP server errors. Significant traffic loss in Microsoft’s network was also observed during the incident,” the network monitoring firm said.
Shortly after 10pm GMT and five hours in, Microsoft engineers were struggling.
“We’re opening all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief” they said. The status updates on X suggested that they had “found a core process on affected servers is becoming overwhelmed, causing capacity loss” – but not why that was happening.
(The wording hints at something triggering CPU spikes in the affected racks.)
As The Stack engineers had tried the trusted “turn if off then and on again” approach – “isolating a portion of infrastructure, restarting affected components, and reintroducing traffic to confirm the services remain stable” before broader testing.
Shortly after 10pm GMT/14:00 PT its engineers said they were “also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.”
There was no indication as The Stack published that the incident was a cybersecurity-related one.
Shortly before midnight GMT/17:00 PT its latest status said “we've begun returning a portion of the affected servers to service, and they're remaining stable so far.”
“We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.” The Stack loves a good post-mortem and we await this one with some interest. More as we have it.
