Facebook outage: Company lost ‘$100million’ during seven-hour blackout

Facebook’s seven-hour blackout that cost the company an estimated $100million in lost revenue alone was made worse because staff were working from home, an insider has claimed. 

The global outage – which hit Facebook, Instagram, WhatsApp and Messenger on Monday – was caused when a faulty update that disconnected its servers from the internet, meaning engineers had to travel to its Santa Clara data center to fix the glitch in-person.

But the repair was delayed, according to one insider who was posting on Reddit, because of ‘lower staffing in data centers due to pandemic measures’.

The glitch also brought down messaging services that remote-working staff use to communicate, so those who knew how to fix the servers couldn’t get that information to the teams inside the data-centre, the insider said.

Also disabled were key-fob entry systems at Facebook’s main campus in Menlo, meaning those who had been WFH but rushed back to the office could not get inside while those already inside were unable to access conference rooms and other areas that required a pass.

‘There are people now trying to gain access to… implement fixes, but the people with physical access is separate from the people with knowledge of how to authenticate the systems and people who know what to actually do, so there is now a logistical challenge,’ the insider said.

The outage was partly to blame for a nose-dive in Facebook’s share price that saw $47billion wiped from its value in its second-worst day ever on the stock market, also driven by a whistleblower testifying about the harms the site does to teenagers in Congress this week. 

Mark Zuckerberg – who lost around $7billion amidst the carnage – has previously vowed to make work from home a permanent part of Facebook, telling staff back in June that ‘anyone whose role can be done remotely can request remote work.’ 

The multi-billionaire said he plans to spend around half his time working remotely in 2022, and predicted that half of his staff could be permanently off-site by 2030.

Facebook’s office are currently open but only to 25 per cent capacity, after plans to open fully by October were pushed back to at least January 2022 amid the spread of the Delta Covid variant. 

A person claiming to be a Facebook employee said on Reddit that high numbers of staff working from home made the problem worse. The account was later deleted

A person claiming to be a Facebook employee said on Reddit that high numbers of staff working from home made the problem worse. The account was later deleted 

Users around the world reported problems with Facebook, Instagram and WhatsApp on Downdetector

 

Twitter founder Jack Dorsey appeared to make light of Facebook's plight on Monday. Responding to a post which appeared to show how the facebook.com domain is for sale as a result of the outage, he jokingly asked: 'How much?'

Twitter founder Jack Dorsey appeared to make light of Facebook’s plight on Monday. Responding to a post which appeared to show how the facebook.com domain is for sale as a result of the outage, he jokingly asked: ‘How much?’

A Facebook staff member reportedly accidentally deleted large sections of the code (pictured) which keeps the website online

A Facebook staff member reportedly accidentally deleted large sections of the code (pictured) which keeps the website online

The above Tweet read: 'So, someone deleted large sections of the routing....that doesn't mean Facebook is just down, from the looks of it....that means Facebook is GONE'

The above Tweet read: ‘So, someone deleted large sections of the routing….that doesn’t mean Facebook is just down, from the looks of it….that means Facebook is GONE’

Facebook shares are down by more than 6 percent from last week as a result of the outage on Monday

Facebook shares are down by more than 6 percent from last week as a result of the outage on Monday

Of the staff who are not currently in the office, it is not clear how many will become permanent remote workers.

But a Facebook executive previously told the Wall Street Journal that the company has approved 90 per cent of WFH requests. The only caveat is that salaries may be cut to reflect the locations where people are actually working, as opposed to where the office is based.

Data centre staff are among those who cannot request a permanent WFH.  

Facebook’s problems began around midday Eastern Time (5pm GMT) on Monday, shortly after its servers were updated, and lasted until around 5.45pm (10.45pm GMT) when the servers came back online. It took several more hours for all users to be able to access Facebook’s sites and apps.

During that time, it is thought that Facebook Inc. – the company which owns Facebook, Instagram, WhatsApp and Messenger – lost between $67million and $102million in direct revenue, based on its hourly earnings across 2020 and estimates of its 2021 hourly earnings based on Q1 and Q2 reports. 

Following Monday’s outage, Zuckerberg issued a personal apology to Facebook users – telling them ‘sorry for the disruption’ while adding: ‘I know how much you rely on our services.’ 

But his message was immediately attacked from all sides, with those who use Facebook business saying he failed to take the issue seriously while casual users accused him of ‘making yourself more important than you are’.

Still others said they had enjoyed the outage, and were planning to spend more time off social media in the future. ‘Life was way simpler without these services,’ wrote one. 

John Graham-Cumming, the chief technology officer of web security firm Cloudflare, said Facebook made a series of updates to its border gateway protocol (BGP) which caused it to ‘disappear’ from the internet. 

The BGP allows for the exchange of routing information on the internet and takes people to the websites they want to access.  

Dane Knecht, senior vice president of the firm, said earlier the Facebook Border Gateway Protocol (BGP) routes had been ‘withdrawn from the internet.’ 

Cybersecurity expert, Kevin Beaumont, wrote on Twitter: ‘This one looks like a pretty epic configuration error, Facebook basically don’t exist on the internet right now. Even their authoritative name server ranges have been BGP withdrawn.’   

Facebook, Instagram and WhatsApp were all brought down for almost seven hours yesterday in a massive global outage. The US tech giant said the problem was caused by a faulty update that was sent to its core servers, which effectively disconnected them from the internet

Whatsapp, Instagram and Facebook Messenger, run on a shared back-end infrastructure, creating a ‘single point of failure’ according to experts.

It wasn’t just the main Facebook apps going down, other services, including Facebook Workplace and the Oculus website were also down. 

Facebook’s Chief Technology Officer, Mike Schroepfer, offered his ‘sincere apologies’ for the outage on Monday afternoon.  The scandal-hit company’s shares had dipped by 5 percent on Monday amid the outage and after a whistleblower went public on Sunday night with claims that the firm prioritises ‘growth over safety’. 

There have been a number of social media outages in recent months, with Instagram going down for 16 hours just last month, and all Facebook platforms going offline in June. 

Twitter founder Jack Dorsey appeared to make light of Facebook’s plight on Monday. Responding to a post which appeared to show how the facebook.com domain is for sale as a result of the outage, he jokingly asked: ‘How much?’ 

The cause of the outage remains unconfirmed and it’s unclear if all are linked but not long before Facebook’s entities went down, entries for Facebook and Instagram were removed from the Domain Name System (DNS) it uses. 

A DNS is essentially an internet directory. Whenever someone opens a link or an app, their device has to search the DNS used by the service they are trying to access to find it and then connect them to it. 

WHAT IS THE DOMAIN NAME SYSTEM AND HOW DOES IT WORK? 

The Domain Name System, or DNS, is the directory of the internet.

Whenever you click on a link, send an email, open a mobile app, often one of the first things that has to happen is your device needs to look up the address of a domain. 

There are two sides of the DNS network: the authoritative side, ie webpages and other content, and the resolver side, devices that are trying to access this content.

Every domain needs to have an authoritative DNS provider, servers which store DNS records. Amazon, Cloudflare and Google are among the bigger names in authoritative DNS server provision. 

On the other side of the DNS system are resolvers. Every device that connects to the Internet needs a DNS resolver. 

By default, these resolvers are automatically set by whatever network you’re connecting to. 

So, for most Internet users, when they connect to an ISP, or a WiFi hot spot, or a mobile network, the network operator will dictate what DNS resolver to use.

The problem is that these DNS services are often slow and don’t respect your privacy. 

What many Internet users don’t realise is that even if you’re visiting a website that is encrypted, indicated by the green padlock in your browser’s address bar, that doesn’t keep your DNS resolver from knowing the identity of all the sites you visit. 

That means, by default, your ISP, every WiFi network you’ve connected to, and your mobile network provider have a list of every site you’ve visited while using them. 

Major DNS providers are Google, Amazon and Cloudflare. It’s unclear if all of the sites and services that went down on Monday use the same DNS or not. 

A similar outage at cloud company Akamai Technologies Inc took down multiple websites in July.

Cloudflare’s Mr Graham-Cumming tweeted on Monday that Facebook accidentally ‘disappeared’ from the internet after making a ‘flurry’ of updates to its BGP – Border Gateway Protocol.   

‘Between 15:50 UTC and 15:52 UTC [4.50-4.52pm UK time] Facebook and related properties disappeared from the Internet in a flurry of BGP updates,’ he said.   

When sites go down because of failures in DNS systems, CloudFare tries to repair them. 

Usman Muzaffar, SVP of engineering at Cloudflare, explained to DailyMail.com: ‘Humans access information online through domain names, like facebook.com and DNS converts it into numbers, called an IP address, computers use. 

‘From what we understand of the actual issue —it is a globalized BGP configuration issue. In our experience, these usually are mistakes, not attacks.

‘Border Gateway Protocol (BGP) is the routing protocol for the Internet. Much like the post office processing mail, BGP picks the most efficient routes for delivering Internet traffic. 

‘Today, the directions for how to get to Facebook’s DNS server’s addresses weren’t available (and seem to still be unavailable). Without being able to contact the DNS servers, visitors trying to reach a Facebook property, like facebook.com, will not get an answer and so the page won’t load.’

According to Reuters news agency, security experts tracking the situation said the outage could have been triggered by a configuration error, which could be the result of an internal mistake, though sabotage by an insider would be theoretically possible.

An outside hack was viewed as less likely. A massive denial-of-service attack that could overwhelm one of the world’s most popular sites, on the other hand, would require either coordination among powerful criminal groups or a very innovative technique.

Shares of Facebook, which has nearly 2 billion daily active users, fell 5.5 per cent in afternoon trading on Monday, inching towards its worst day in nearly a year. 

It means that the company’s founder Mark Zuckerberg – who owns around 14 per cent of the firm – has seen his wealth plummet by nearly $7billion in a matter of hours, Bloomberg reported. 

Some users of UK phone network EE were also reporting that they were having difficulty accessing mobile internet services. However, the firm told MailOnline that there were no problems with the network.

Cyber security specialist Jake Moore said there is a ‘chance’ the issue affecting the firms could be related to a cyber attack.

He said: ‘There have been many reports and I’m struggling to find out exactly what has happened- I’m reading it could be DNS related, which means there is an issue with the connection not knowing where to go to your device.

‘It could well be a human error or a software bug lurking in the shadows but whatever it is Facebook needs to do its best to mitigate the problem of causing more panic about this.

‘The biggest problem is fears over a cyber attack but as we saw from Fastly in the summer I would hedge my bets on that not being the case as we’re talking about one of the biggest companies in the world, but there’s always a chance.’

Apologising on Twitter for the outage, Mr Schroepfer said: *’Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible.’

Facebook was already in the throes of a separate major crisis after whistleblower Frances Haugen, a former Facebook product manager, provided The Wall Street Journal with internal documents that exposed the company’s awareness of harms caused by its products and decisions. 

Haugen went public on CBS’s ’60 Minutes’ program Sunday and is scheduled to testify before a Senate subcommittee Tuesday.

Haugen had also anonymously filed complaints with federal law enforcement alleging Facebook’s own research shows how it magnifies hate and misinformation and leads to increased polarization. It also showed that the company was aware that Instagram can harm teenage girls’ mental health.

The Journal’s stories, called ‘The Facebook Files,’ painted a picture of a company focused on growth and its own interests over the public good. Facebook has tried to play down the research. 

Former Deputy Prime Minister Nick Clegg, the company’s vice president of policy and public affairs, wrote to Facebook employees in a memo Friday that ‘social media has had a big impact on society in recent years, and Facebook is often a place where much of this debate plays out.’

Earlier on Twitter, Facebook communications executive, Andy Stone said they were aware some people were having trouble accessing Facebook apps and products.

‘We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience,’ the executive said in a tweet.

Soon after the first report came through, the hashtag #facebookdown was trending on Twitter, with users worldwide reporting issues connecting. 

The hashtag #instagramisdown and ‘WhatsApp’ were both also trending on Twitter, with a number of users saying they checked their internet connection when they couldn’t get on Facebook. 

Instagram comms tweeted: ‘Instagram and friends are having a little bit of a hard time right now, and you may be having issues using them. Bear with us, we’re on it!’

They're some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon. Above: The reports of Facebook outages reported on DownDetector

They’re some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon. Above: The reports of Facebook outages reported on DownDetector

Down Detector also showed how problems with Whatsapp began being reported just before 5pm on Monday

Down Detector also showed how problems with Whatsapp began being reported just before 5pm on Monday

Facebook Messenger's outage was also reported on DownDetector at a similar time on Monday

Facebook Messenger’s outage was also reported on DownDetector at a similar time on Monday

At 11.33pm on Monday, Facebook tweeted to apologise about the global outage of its services. They added that they were 'happy to report' that they were coming back online

At 11.33pm on Monday, Facebook tweeted to apologise about the global outage of its services. They added that they were ‘happy to report’ that they were coming back online

On Twitter, Facebook communications executive, Andy Stone said they were aware some people were having trouble accessing Facebook apps and products

On Twitter, Facebook communications executive, Andy Stone said they were aware some people were having trouble accessing Facebook apps and products

The various Facebook-owned platforms, including WhatsApp, Instagram and Facebook itself, took to Twitter to explain the issues and say they were working on a solution

The various Facebook-owned platforms, including WhatsApp, Instagram and Facebook itself, took to Twitter to explain the issues and say they were working on a solution

RECENT FACEBOOK OUTAGES 

Last month, a technical issue with Facebook owned Instagram caused an outage that plagued users around the world for 16 hours.

Problems started just after 8am on Thursday. About 18 hours later, at 2am on Friday, Instagram announced the problem had been fixed.

However, the last time Facebook, Instagram and WhatsApp went down at the same time was in June. 

In June more than a thousand people in countries including the United States, Morocco, Mexico, Bolivia and Brazil reported outages.

There were also two Facebook platform outages in March, with Instagram down on March 30, and all three down on March 19. 

WhatsApp tweeted: ‘We’re aware that some people are experiencing issues with WhatsApp at the moment. We’re working to get things back to normal and will send an update here as soon as possible. Thanks for your patience!’

Even Oculus, the virtual reality gaming platform owned by Facebook was having problems, with one user describing their headset as being ‘like a paperweight’.

Oculus tweeted: ‘We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.’ 

Every time Facebook and Instagram are down, it draws people to Twitter. 

On Monday, one user shared a meme of Homer Simpson jumping from his house to Moe’s bar, with the Twitter logo over the door.

Even Google got in on the action, tweeting: ‘Everyone going to Google to check if Instagram is down.’ 

There were multiple jokes along the same lines, with one showing a fast track race and the caption: ‘Me and my friends running to twitter to see if fb, whatsapp and insta are down.’ 

It is unclear what has caused the issue, although it has disrupted all Facebook owned platforms, including the Oculus Virtual Reality gaming website and Facebook Workplace.

NetBlocks, which tracks internet outages, tweeted: ‘Facebook, WhatsApp, Instagram and Messenger are currently experiencing outages in multiple countries.’

Adding that the ‘incident not related to country-level internet disruptions or filtering.’

When attempting to visit Instagram using a desktop web browser, it gives up a ‘5xx Server Error’ and Facebook simply says ‘this site can’t be reached.’ 

Twitter users also took aim at Mark Zuckerberg by joking that he was trying to fix the problems himself

Twitter users also took aim at Mark Zuckerberg by joking that he was trying to fix the problems himself

Another tweet depicted Zuckerberg as a workman recovering an overturned car. The post above the image read: 'Mark Zuckerberg watching his team fix Whatsapp, Instagram and Facebook'

Another tweet depicted Zuckerberg as a workman recovering an overturned car. The post above the image read: ‘Mark Zuckerberg watching his team fix Whatsapp, Instagram and Facebook’

Twitter users took to the platform to make light of the situation and highlighted how millions of people were flocking to it in the absence of Facebook, Instagram and Whatsapp

Twitter users took to the platform to make light of the situation and highlighted how millions of people were flocking to it in the absence of Facebook, Instagram and Whatsapp 

There were multiple jokes along the same lines, with one showing a fast track race and the caption: 'Me and my friends running to twitter to see if fb, whatsapp and insta are down'

There were multiple jokes along the same lines, with one showing a fast track race and the caption: ‘Me and my friends running to twitter to see if fb, whatsapp and insta are down’

Twitter made light of their rivals' problems on Monday by posting a tweet which read: 'hello literally everyone'

Twitter made light of their rivals’ problems on Monday by posting a tweet which read: ‘hello literally everyone’

UK network Vodafone made light of Facebook's plight by joking that they had 'top men' working on the issue

UK network Vodafone made light of Facebook’s plight by joking that they had ‘top men’ working on the issue

The last major outage of Facebook platforms was in June 2021, when people in the US, Morocco, Mexico and Brazil all reported not being able to connect. 

However, there were also problems last month, when Instagram went down for a whopping 16 hours. 

Jake Moore, cybersecurity specialist at ESET said outages are increasing in volume and are becoming increasingly harder to predict.

He said initially, a major problem with a website or app can point towards a cyber attack – but that can add to confusion and be misleading.

‘With recent issues such as what happened with Fastly [the web service platform that saw a major global outage on June 8] it highlights the power of an undiscovered software bug or even human error,’ Moore explained.

Soon after the first report came through, the hashtag #facebookdown was trending on Twitter, with users worldwide reporting issues connecting

Soon after the first report came through, the hashtag #facebookdown was trending on Twitter, with users worldwide reporting issues connecting

‘Although these are increasing in frequency and require more failsafes in place, predicting these issues is increasingly more difficult as it was never thought possible before. Luckily, most outages only last under an hour.’

This latest outage highlights the major issues with using centralised systems, according to Matthew Hodgson, Co-founder and CEO of Element and Technical Co-founder of Matrix.

‘The ongoing outage of WhatsApp, Instagram and Facebook (including Facebook Messenger and Facebook Workplace) highlights that global outages are one of the major downsides of a centralised system,’ he said.

Centralised apps, like having a single back end for Facebook products, means putting ‘all the eggs in one basket,’ Hodgson explained.

‘Decentralised systems are far more reliable. There’s no single point of failure so they can withstand significant disruption and still keep people and businesses communicating.’

NetBlocks, which tracks internet outages, tweeted: 'Facebook, WhatsApp, Instagram and Messenger are currently experiencing outages in multiple countries'

NetBlocks, which tracks internet outages, tweeted: ‘Facebook, WhatsApp, Instagram and Messenger are currently experiencing outages in multiple countries’

They're some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon

They’re some of the most popular social media apps around the world, but it appears that Facebook, Instagram, WhatsApp and Facebook Messenger have all crashed this afternoon

***
Read more at DailyMail.co.uk