Microsoft says a sweeping global outage of Microsoft 365 and other services like Defender and Intune was “caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.”
1) A “latent defect” in a July patch for Microsoft 365 networking gateway servers that was not caught during load testing, which focused on “weaker hardware SKUs with fewer cores” – but which triggered issues on newer hardware with more cores.”
(Notably, although Microsoft did not tie the incident to this, October 2026 Windows Server updates from Redmond caused similar downstream borkage for customers. Redmond EVP for Copilot, Agents, and Platform Charles Lamanna pointed however to a July fix “prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments,” he said.)
See also: N2, N1, N0ooo… How Google Cloud fell over in London
2) A “surge in load” that triggered the bug.
Posting on X, early on Monday, Lamanna said: “Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability… fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online as part of “normal operations” (Microsoft has promised a final Post-Incident Review (PIR) later this week – perhaps that will be more revealing on the specifics.)
3) Microsoft had “nearly 14% of server capacity for the networking gateway…out of service due to an unrelated issue.” Requests built up, failure cascaded from server to server as the same amount of load was spread across fewer servers: “In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.”
A separate issue, said Lamanna, meant “normal bulkheads” did not stop the cascading failure” – and because the main code defect was a “latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change,” he said. (Redmond’s preliminary report is gated here.)
More later this week when The Stack has the full post-mortem – including, hopefully, more detail on why 14% of networking server capacity was offline and why those "normal bulkheads" failed.