Microsoft blames massive Microsoft 365 outage on maintenance bug - BleepingComputer

What the report says

BleepingComputer reported that Microsoft attributed a major Microsoft 365 and Azure disruption on Thursday, July 23, to a flaw in an automated system used to process network maintenance requests. According to the report, the incident began at 10:44 a.m. ET and mainly affected customers reaching Microsoft 365 through network infrastructure tied to the West US Azure region. Downdetector showed 2,403 reports by 11:11 a.m. ET, compared with a normal baseline of 29, with SharePoint making up most complaints listed in the article.

Microsoft tracked the Microsoft 365 issue as incident MO1437424. BleepingComputer said affected services included OneDrive, SharePoint Online, Teams chat, the Microsoft 365 Admin Center, Power Automate, Copilot Chat, Loop, Fabric, Power BI, Power Apps, Copilot Studio, Windows 365 and Microsoft Defender. Some users saw intermittent access, degraded chat functions, slow or failed admin center loading, and delays or failures in cloud-based actions.

The publisher cited Microsoft’s preliminary Azure post-incident review, which said routine maintenance in the West US region was meant to isolate specific network paths. A bug in the request conversion process wrongly included additional network devices, causing IP routes to be removed from more devices than planned between a West US datacenter and Microsoft’s wide-area network. Traffic inside the region was not affected, but traffic entering or leaving the region saw connectivity failures and latency.

Microsoft first tried rerouting traffic, then began rolling back the networking change at 1:45 p.m. ET. BleepingComputer reported that Microsoft completed the rollback at 2:26 p.m. ET, resolved the Microsoft 365 incident, and later said all affected Azure services had recovered by 3:41 p.m. ET. Microsoft said it is reviewing safety checks and automated maintenance processes and expects to publish a final review after its investigation.