Authorize.net, along with several other major online companies, was down for up to 15 hours after a fire at the Fisher Plaza, a “world class” datacenter in Seattle, Washington Thursday.
Sometime around 11PM on July 2 an electrical short in a parking garage beneath Seattle’s Fisher Plaza caused a small fire that set off fire alarms and sprinklers. Fisher Plaza’s datacenter is home to numerous online companies including Authorize.net, one of the largest online payment gateways in the world.
The fire department cut power to the building and evacuated all personnel. The sprinklers soon flooded the datacenter’s generator rooms, leaving the building without backup power.
The outage affected hundreds of thousands of e-commerce merchants who rely on Authorize.net to process credit card payments for their websites. Some estimate the losses in the hundreds of millions.
The Topic Trends on Twitter
Almost immediately after the outage began, people started posting to Twitter, a popular micro-blogging service, about the outage. Merchants vented about their frustrations and anger as they sat helpless, unable to process transactions for over 15 hours…
@xemion: Wonder what today’s Authorize.net outage will do to PayPal’s bottom line numbers today.
@dancubed: Fire takes down Authorize.net, halting e-commerce for many. I find it amazing no geo redundancy. Amateurs.
@LinhN: damn, authorize.net likely lost a bunch of customers
@paulpacek: Authorize.net mega fail this morning. Can I send you a Honda generator? I would like to get my business back up and running asap.
Authorize.net Breaks Silence
Soon, major media outlets were also covering the outage. At around 11 AM EST, Authorize.net setup an official Twitter account at @AuthorizeNet in order to respond to the massive conversation going on…
While some commend Authorize.net for using Twitter to communicate during the outage, I’d like to remind everyone that they waited 12 hours before they responded. The service was back up shortly after the account was setup.
I think this was a major failure in communication, since for almost 12 hours merchants had no way to communicate with the company about the outage that was costing them millions of dollars in lost revenues. Authorize.net was not answering phones and their website and services were down. Merchants were completely in the dark.
Where Authorize.net REALLY Failed
This incident really showed how unprepared Authorize.net and Fisher Plaza were for any kind of emergency. With all of the networking equipment and the power utilization of a datacenter, a fire is the most likely disaster to occur. Where were the failover systems? Why were sprinklers used instead of an alternative method such as Halon gas, that would not have damaged servers and equipment?
So much for a “world class” datacenter… Fisher Plaza sounds more like a bunch of servers set up in a garage. What’s more, the Fisher Plaza datacenter had a similar catastrophe in 2006. Why did they not learn from their mistakes? Could this be considered criminal negligence?
Authorize.net is just to blame, if not more, because they were undoubtedly aware of the 2006 outage and yet still continued to host their servers in the Fisher Plaza facility. Without adequate contingency plans in place, it was just a matter of time before another major outage.
With hundreds of thousands of online merchants relying on them for payment processing, it’s incomprehensible to me that Authorize.net would put all their eggs in one basket. Why was there no geo-redundancy? Authorize.net should have had operations in another datacenter, with mirrored servers, ready to go in the event of a catastrophic failure. Earthquakes, tornadoes and fires happen. Be prepared.
What can YOU do when there is an outage?
As an e-commerce merchant, what can you do to prepare for an outage? When you rely on third party vendors such as a payment gateway or shipping service, you need to have a contingency plan in place.
Manual Processing
When Authorize.net went down, hundreds of thousands of websites began giving error messages when customers tried to place orders. Some, like myself, turned off online processing. We captured information (not including CVV values, which would be against Visa/MC regulations) for offline processing later.
This allowed us to continue to operate, somewhat transparently to the customer. There is a chance that customers will enter their information incorrectly, or you’ll get some declined transactions but those can be handled case-by-case after processing resumes.
Backup Processor
Another option is to have a backup processor (or shipping carrier, etc) that you can turn to if/when your primary provider fails. There might be a slight configuration change necessary with your e-commerce software, but often times it would only take moments to switch over to an alternate processor.
Having an account with a second processor might cost $20-30 in fees every month, but that’s a valuable insurance policy against catastrophe. For just a few hundred dollars a year, you could keep from losing tens of thousands of dollars in lost revenues.
Tell Your Story
What is your Authorize.net downtime story? Tell us how the downtime affected you or your customers in the comments below.
[…] one of the largest payment gateways in the world, was down for 15 hours after a fire at Fisher Plaza, the Seattle, WA based datacenter where they were […]
We’re a new service launching on Monday with, guess what, Authorize.net as both our payment gateway and merchant account provider.
We are stuck in test mode until we can confirm our payment and merch accounts are, in fact, working — so far, we get confirmation via test CC#’s and “invalid merchant account” using valid CC#’s.
Where Authorize.net is failing for us?
After the worst 12-36 hours in the corporate history, they aren’t available this weekend to take support calls? Really?
Any alternatives to Authorize.net that can have us up and running with a G’way and Merchant acct by Monday? Contact me directly at James at ShareOnce dot com…and yes, I’ll be working all weekend. :)
[…] The E-Commerce Blog | Fire at Authorize.net Datacenter has … By Brandon Eley When Authorize.net went down, hundreds of thousands of websites began giving error messages when customers tried to place orders. Some, like myself, turned off online processing. We captured information (not including CVV values, … The E-Commerce Blog – http://www.theecommerceblog.com/ […]
Did they ever hear of DRP? I am sure they will overcome it; after all, we are speaking of the biggest provider of gateway services in the US. Just read about them in http://www.creditcardprocessing-r-us.com . Looks like there is no real substitute for their services, but there is a serious lesson they should learn here.
(disclaimer: I work for a company in the stored value sector of the market)
I am the IT manager at this company. While we’re smaller than authorize.net we’re EONs in front of them in terms of redundancy and contingency plans. While we have been working with getting automatic zero-interuption failover to our second facility I couldn’t imagine a scenario where we’d have a downtime of more than an hour, and that’s being generous. And for even that to happen it would have to be a catastrophic failure like the main datacenter exploding and disintegrating all servers.
Some more bashing of the datacenter itself. The one we’re located at uses a dry chemical (they purposely don’t use the haylon gas because of the safety concerns for people that may be in the datacenter at the time and the dry chemical can be directed to only the rack that’s caught fire). Also the generators are located in a room next to the building, not in the basement like this sounds. And the cause of the sprinklers in fisher plaza was due to a spark in the basement… ANYTHING happening outside of the datacenter shouldn’t affect it.
I guess they say people learn from mistakes and I’ve seen that happened with lots of large companies, perhaps they’ll learn their lessons to avoid this from happening in the future.
I know this was a few years ago and very sad indeed. Seems that hey have bounced back and still appear to be the leading merchant gateway in the industry.