Facebook Outage

[WrongVerb]

A computer error has caused Facebook to disappear from the internet.

Facebook is down, along with Instagram, WhatsApp, Messenger, and Oculus VR

Just as Facebook’s Antigone Davis was live on CNBC defending the company over a whistleblower’s accusations and its handling of research data suggesting Instagram is harmful to teens, the company’s entire network of services suddenly went offline. On Twitter, Facebook communications exec Andy Stone says, “We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.”

A peek at Down Detector (or your Twitter feed) reveals the problems are widespread. While it’s unclear exactly why the platforms are unreachable for so many people, their DNS records show that, like last week’s Slack outage, the problem is apparently DNS (it’s always DNS). Cloudflare senior vice president Dane Knecht notes that Facebook’s border gateway protocol routes — BGP helps networks pick the best path to deliver internet traffic — have been “withdrawn from the internet.”

Other info...

Reddit r/Sysadmin user that claims to be on the "Recovery Team" for this ongoing issue:

>As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.

User is providing live updates of the incident here: https://www.reddit.com/r/sysadmin/co...dyv/?context=3

Sheera Frenkel
@sheeraf
Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.
2:51 PM · Oct 4, 2021·

[Bob Sheed]

[kaldaniels]

Just checking in on our mods.

Have you been able to enter Redszone HQ with your retina scan today?

[Boston Red]

Facebook stock is down 4.89% today. The NASDAQ was down 2.14%, so the Facebook drop wasn't as bad as I thought it might be.

[WVRed]

Looks like Google is having problems now.

[SteelSD]

Why is it that I'm picturing millions of Americans furiously alternating between unplugging their computer and plugging it back in...

...repeatedly...

...all day long...

[cumberlandreds]

Just checking in on our mods.

Have you been able to enter Redszone HQ with your retina scan today?

No problems here.

[Roy Tucker]

Hey, what’s this button do?

[BernieCarbo]

Roy, can you hypothesize of what might have happened?

[RedTeamGo!]

Roy, can you hypothesize of what might have happened?

I’ve heard one of the systems admins “didn’t like the way the company was heading” and shut it down

[BernieCarbo]

That would be very unusual if one guy had that power. I guess it’s feasible, but that is quite a tightrope that Facebook would be walking.

[WrongVerb]

That would be very unusual if one guy had that power. I guess it’s feasible, but that is quite a tightrope that Facebook would be walking.

If they updated something, I've no doubt their process included some type of peer review. It's also unlikely that a developer was also the person pushing the changes to ops. Sounds like a failure at multiple levels.

[Todd Gack]

I was really hoping this social media outage would've lasted for a much longer time, we all suddenly realize how much happier and well-adjusted we'd have become.

Unfortunately, we're not so lucky.

[WVRed]

I was really hoping this social media outage would've lasted for a much longer time, we all suddenly realize how much happier and well-adjusted we'd have become.

Unfortunately, we're not so lucky.

Twitter and TikTok still worked.

[Roy Tucker]

Roy, can you hypothesize of what might have happened?

Being in this business (albeit much smaller scope), I was curious as to how this can happen to someone like Facebook. I know that we have a very rigorous process for promoting changes up to our production environment and woe be to you if you don't follow the process. But we are all imperfect carbon based units and people skip steps and think "I can just make this one change and it won't affect anything". Believe me, it's easy to do and I've gotten caught once on it (and once is all the chances you get in our company, I've seen people walked out that day over it). It has to be taken very seriously.

It was a Border Gateway Protocol (BGP) issue. BGP is how your internet-facing servers speak to one another. A very telling statement (found in the Facebook link below) was "During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally.".

Haha. Oh jeez. Holy moly. Oh god. My worst nightmare. There are some production changes that I have made in the past where I realize that this is a Big Change and I damn well better be scrupulously 1000% correct and really know WTF I'm doing. Like I tell our junior engineers and developers, with great power comes great responsibility. But like I said, all of our changes are made in lower environments thoroughly tested out before being promoted up. So for those big changes, I *know* what the impact is. But in this case, it was a Facebook network engineer (I assume ex-Facebook now) that roguely and out of band did this command without completely understanding the ramifications. And man did they screw up big time. Human error.

So once the Facebook DNS entries are removed from the internet backbone routers via BGP, you're in really deep yogurt. Your urls are removed from the internet and all of those changes are propagated out everywhere. And I mean *everywhere*. You are effed big time. You can't even access your own systems via VPN since your service provider doesn't know anything Facebook exists, you have to drive into the data center to get on machines there, and everyone is running around with their hair on fire. We have had one catastrophic outage at my company and it was no fun at all. At least with us, we routinely practice our DR exercises so we knew what to do. But stuff hits the fan big time.

So, tl;dr, human error. Detail below from internally Facebook and externally from Cloudfire. Interesting reads.

https://engineering.fb.com/2021/10/0...utage-details/

https://blog.cloudflare.com/october-...cebook-outage/