Facebook is back online after massive global outage

No, it wasn’t your slow internet. Facebook, WhatsApp, and Instagram all went down in Sri Lanka and for many around the world. The Verge reported that even Oculus and Facebook’s own internal sites were affected.

At the time facebook’s Andy Stone tweeted “We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.” A similar message was shared across all of Facebook’s Twitter channels.

What happened

While Facebook is yet to confirm the exact reason for the outage, chances are it’s yet another DNS issue. Dane Knecht, Senior Vice President at Cloudflare mentions that Facebook’s Border Gateway Protocols (BGP) routes have been withdrawn from the internet. For those wondering what BGP is, it’s what helps establish the best path to send data from one device to another until it reaches the final destination. Think of it as the postal service of the internet.

Journalist Brian Krebs citing a source claims that the issue was a routine BGP update gone wrong. This meant all the necessary DNS routing information that other networks need to find Facebook’s sites, were gone.

Additionally, the update prevented remote users from reverting changes while people with physical access didn’t have network access. In other words, they were blocked from both ends from reversing the changes.

Surging traffic on the rest of the internet

As Facebook, Instagram, and even its own status page went offline, many flocked to Twitter and other services to fill the gap that Facebook left. Messaging app Signal claims that millions of new people already signed up as a result of the Facebook outage.

But the situation meant that other websites took the hit due to the sudden traffic spike. Cloudflare had to deploy additional resources just to keep with the traffic of people repeatedly trying to reload Facebook and Instagram. One down detector site “Is It Down Right Now” also found it difficult to keep up owing to the massive influx of inquiries.

The aftermath

By now, all of Facebook’s platforms are back online. But the outage was so problematic that it nearly broke all of Facebook’s internal systems. According to The Verge, several employees had to resort to their work-provided Outlook accounts, Google Docs, and Zoom for communication while others were blocked entry to office buildings altogether.

Since a remote fix wasn’t possible, Facebook engineers were sent to one of its main US data centers in California to restore the service. It took a near six hours for FAcebook’s services to come back on completely. Following successful restoration, Mike Schroepfer, CTO at Facebook, publicly apologized for the disruption.

This isn’t the first time that Facebook went off the grid. But the outage still proved to be a costly one for the tech giant with Fortune citing that Facebook lost nearly $100 million in revenue.

Issuing a statement on the incident, Facebook clarifies that the root cause of the outage was a faulty configuration update. It further mentions that the company has “no evidence that user data was compromised as a result of this downtime.”

[05/10/2021 01:04 AM]: Facebook sites continue to be offline

[05/10/2021 05:07 AM]: Facebook sites are finally back online after a near six-hour outage. Article updated with added context, along with comments from Facebook CTO

[05/10/2021 08:05 AM]: Article updated with statement from Facebook