Facebook blames a server configuration change for yesterday’s outage

https://postimg.cc/N5wmBjNc

Facebook has finally released its assessment of what was responsible for yesterday’s massive outage of its Instagram, Messenger, and core Facebook service.

Contrary to the thesis circulating yesterday that it was a route leak, the company said that a server configuration change was to blame.

Millions of people were impacted by the outage yesterday, which had users taking to Twitter (and our tips inbox) to vent their frustration or share theories about what was happening inside the company’s networks.

The Facebook outage was the largest ever recorded by the service Downdectector.

“By duration, this is by far the largest outage we have seen since the launch of Downdetector in 2012,” said Tom Sanders, co-founder of Downdetector. “Our systems processed about 7.5 million problem reports from end users over the course of this incident. Never before have we such a large scale outage.”

Network monitoring firms first began specualting about the cause of the outage yesterday.

“The cause would appear to be internal rather than a network or Internet delivery issue–for example we saw ‘500 internal server errors’ from Facebook,” a ThousandEyes spokesperson wrote in an email. “Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.”

Netscout, which as we reported initially claimed the outage was due to a BGP route leak, later retracted that explanation.