Uncategorized

Microsoft Teams went down around the world for over eight hours — it’s recovering

Illustration of the Microsoft wordmark on a green background
Illustration: The Verge

Was it a joyous outage where you got to relax, or did something mission-critical go wrong? Either way, “a portion” of Microsoft Teams went down Friday for over eight hours — with outages in North America, South America, Europe, the Middle East, and Africa, according to status updates from Microsoft’s official account.

At 8:17PM ET Friday, Microsoft tweeted that the worst is over, but that the company’s still mopping up, with a handful of issues still ongoing.

So yeah, if you couldn’t communicate with your colleagues today via Microsoft’s biz comm platform, that’s probably why. We’d seen reports of users not being able to log into Teams at all, while others saw missing messages, missing attachments, delays, and more.

At this point, Microsoft says you should be able to log in, load messages, view media, and access call recordings — but that anonymous users may still be unable to join meetings, you may still see delays, you may have trouble rejoining meetings, or loading Copilot history, and you probably shouldn’t reboot your computer if you need to log into Teams since that was an issue too.

At least not until Microsoft gives the all-clear, anyhow.

Here’s the company’s overall guidance now:

We’re seeing significant improvements with many of the Teams features affected by this incident as we continue to apply mitigations and optimize traffic. We’re evaluating the remaining impact scenarios and are developing fixes and workstreams to address all lingering impact associated with this event.

The spike in problem reports at DownDetector suggested this outage had been going on for at least three hours at the time we originally published our story, and Microsoft officially recognized the issue at around 11:45AM ET. The company said it identified “a networking issue impacting a portion of the Teams service,” and began failovers to resolve the problem.

Four hours later, though, Microsoft reported that “Our failover operation did not provide immediate relief to all end users in North and South America regions.” While the DownDetector spike has subsided, it’s not clear whether that’s because services are substantially restored or because there’s no point complaining anymore.

Here’s how far Microsoft got by the eight-hour mark:

We’re continuing to apply mitigations across the affected infrastructure and our telemetry is showing additional improvement in the user experience, though many customers are still affected by this issue. We’re also working to apply fixes to address individual affected Teams features in parallel while our broader remediation strategy is ongoing. We’re evaluating any and all additional workstreams that will allow us to reduce the impact to those customers that are still affected.

If you’ve got access to a Microsoft Teams admin console, you can check the status at TM710344, though I imagine most admins already know that by now.

For hours after the outage began, Microsoft’s status page was all green, suggesting Microsoft Teams was fine, but it updated with this description shortly before 2PM ET:

Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren’t limited to:

– Users performing a cold boot may not able to log into teams and will see an “oops” page.

– Users logging in or unlocking their devices after some time may see missing messages.

– Users may fail to load messages in channels and chats

– Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)

– Some messages may experience delays being sent

– Call Recordings might take longer to appear in user’s OneDrive for Business and SharePoint Online

– Bots may be unable to download attachments

– Sending and receiving read receipt notifications may be delayed

Current status: Our monitoring systems alerted us to an issue where users may experience various impact scenarios in Microsoft Teams, outlined in the More info section above. We’ve identified that a portion of the Microsoft Teams service is experiencing a networking issue. We’ve completed a failover in the Europe, Middle East, and Africa (EMEA) region and telemetry is showing improvement. We’re continuing work to failover service traffic in all affected regions to remediate impact.

Next update by: Friday, January 26, 2024, at 9:00 PM UTC

IT admins saw a similar message in their dashboards around 1PM ET, according to an r/sysadmin user:

Microsoft Teams had a four-hour outage almost exactly a year ago. It had a pair of large outages in 2021, and ones in 2020 and in 2019, too, but this eight-hour-plus outage may take the cake — depending on how many people were affected, anyhow.

Update, 8:00PM ET: Added that the outage is still ongoing, and Microsoft’s latest update.

Update, 9:04PM ET: Added that the outage seems to be mostly over, but a number of specific issues remain.