Authorize.net, along with several other major online companies, was down for up to 15 hours after a fire at the Fisher Plaza, a “world class” datacenter in Seattle, Washington Thursday.
Sometime around 11PM on July 2 an electrical short in a parking garage beneath Seattle’s Fisher Plaza caused a small fire that set off fire alarms and sprinklers. Fisher Plaza’s datacenter is home to numerous online companies including Authorize.net, one of the largest online payment gateways in the world.
The fire department cut power to the building and evacuated all personnel. The sprinklers soon flooded the datacenter’s generator rooms, leaving the building without backup power.
The outage affected hundreds of thousands of e-commerce merchants who rely on Authorize.net to process credit card payments for their websites. Some estimate the losses in the hundreds of millions.
The Topic Trends on Twitter
Almost immediately after the outage began, people started posting to Twitter, a popular micro-blogging service, about the outage. Merchants vented about their frustrations and anger as they sat helpless, unable to process transactions for over 15 hours…
@xemion: Wonder what today’s Authorize.net outage will do to PayPal’s bottom line numbers today.
@dancubed: Fire takes down Authorize.net, halting e-commerce for many. I find it amazing no geo redundancy. Amateurs.
@LinhN: damn, authorize.net likely lost a bunch of customers
@paulpacek: Authorize.net mega fail this morning. Can I send you a Honda generator? I would like to get my business back up and running asap.
Authorize.net Breaks Silence
Soon, major media outlets were also covering the outage. At around 11 AM EST, Authorize.net setup an official Twitter account at @AuthorizeNet in order to respond to the massive conversation going on…
While some commend Authorize.net for using Twitter to communicate during the outage, I’d like to remind everyone that they waited 12 hours before they responded. The service was back up shortly after the account was setup.
I think this was a major failure in communication, since for almost 12 hours merchants had no way to communicate with the company about the outage that was costing them millions of dollars in lost revenues. Authorize.net was not answering phones and their website and services were down. Merchants were completely in the dark.
Where Authorize.net REALLY Failed
This incident really showed how unprepared Authorize.net and Fisher Plaza were for any kind of emergency. With all of the networking equipment and the power utilization of a datacenter, a fire is the most likely disaster to occur. Where were the failover systems? Why were sprinklers used instead of an alternative method such as Halon gas, that would not have damaged servers and equipment?
So much for a “world class” datacenter… Fisher Plaza sounds more like a bunch of servers set up in a garage. What’s more, the Fisher Plaza datacenter had a similar catastrophe in 2006. Why did they not learn from their mistakes? Could this be considered criminal negligence?
Authorize.net is just to blame, if not more, because they were undoubtedly aware of the 2006 outage and yet still continued to host their servers in the Fisher Plaza facility. Without adequate contingency plans in place, it was just a matter of time before another major outage.
With hundreds of thousands of onlin