What Will It Take to Get the Internet Running, After Sandy?
Hot water and infrastructure weren’t the only things affected by Monday’s storm. Hurricane Sandy took down New York City-based media companies Huffington Post, Gawker and BuzzFeed.
The three brands house their servers with Datagram, located beside Battery Park, where the flooding hit a record high of about 14 feet.
When Con Edison lost power to Lower Manhattan, Datagram’s emergency systems kicked on to maintain the data centers’ power.
According to Datagram’s website, the basement began to flood around 5 p.m. during the storm. The water overwhelmed the pumps, which fuel backup generators to keep the servers online.
“Due to electrical systems being underwater, the building was forced to shut down to avoid fire and permanent damage,” the company wrote.
Thus, in one of the most crucial news moments of 2012, media companies had to make quick decisions.
Gawker is still running “emergency” pared-down sites throughout all of its properties, including Gizmodo, Jezebel and Kotaku.
BuzzFeed chose to continue updating via social platforms. The company wrote on Tumblr that “elements” of the site would return online, thanks to Akamai, a content delivery network that hosts content on servers all over the world.
The engineering team then rebuilt all of BuzzFeed’s data using Amazon cloud storage, with one developer working overnight — even after a tree fell through his roof.
The Huffington Post was also up all night recovering data. The news site hosts its data across the country, but its main data center lives at Datagram — with a backup data center in Newark, N.J.
The tech team at Huffington Post scrambled to switch to the emergency data center in Newark; however, they ran into issues there as well, as HuffPo‘s CTO John Pavley pointed out. The editorial staff relied on social platforms to update readers with new information during the storm.
Up to Tuesday morning, The Huffington Post was accessible through a temporary site. It is currently back up and running.
The storm may be over, but the Internet hasn’t spotted a rainbow just yet. U.S. Internet traffic was rated 67 out of 100 Tuesday, with several routers across the country reporting 100% packet loss, according to the Internet Traffic Report (ITR). By Wednesday, the ITR still reported New York City at 100% packet loss.
Other server maintenance companies like Fog Creek and Squarespace are located not too far from Datagram in Manhattan. Though they did not experience as unfortunate a fate as Datagram, it’s literally taking superhuman strength to keep these servers running.
“We preemptively shifted to backup power around 4:00 p.m. yesterday, predicting that Con Edison would be shutting off power in evacuation zones,” wrote Squarespace CEO, Anthony Casalena on Tuesday. “Given the nature of the flooding, this situation escalated greatly, submerging our reserve fuel in the basement, shutting off the elevators and damaging the pumps required to get this fuel to the generator on the 17th floor.”
Pumping water out of the basement and getting the elevators back up will require a significant amount of time, which the generators don’t have. Instead, the teams are focusing on fueling the pumps.
Volunteers from Fog Creek, Squarespace and PEER 1 have been hauling drums of diesel fuel up 18 flights to keep servers running.
The group of volunteers has been labeled the “bucket brigade.” More than 30 volunteers take shifts, and PEER 1 tweeted that reinforcements from Toronto, Montreal and Atlanta were on the way.
Fog Creek updated its blog on Wednesday, saying it was “highly optimistic.” According to the company, the services are continuing to run regularly and the water in the basement is being removed at about one foot per hour.
Some of those affected by lack of Internet, power and cell service have found employees or friends fortunate enough to have those amenities. Many, especially in New York, are walking miles just to charge their devices at Starbucks or get work done wherever they can.
Are New Yorkers crazy, admirable — or a little bit of both?
It’s unclear exactly when the Internet world will go back to normal. Some have hinted it may take a week, but it’s difficult to place a hard number on something as unpredictable as the Internet.
November 1, 2012