VideoTo understand how Hurricane Katrina is changing the way Corporate America thinks about disaster recovery, look no further than the computer-services company in New Orleans that located its data center to the third floor of its building to preclude any possible risk of flood damage. "I salute them for planning," says Hank Chase, director of homeland-security programs for management consulting firm Smart and Associates LLP, in Devon, Pennsylvania. "They knew they were in an area subject to flooding." Unfortunately, he says, no one imagined the building could be inundated by floodwater 23 feet deep, which is what Katrina delivered when it broke through the city's levees in August. The company's data center was destroyed. "They had no backup systems, no redundant infrastructure whatsoever," observes Chase. "The last I heard, they're down completely — out of business temporarily."
Of all the lessons learned from Hurricane Katrina, the most important is this: companies must rethink the definition of "worst case." "Based on public information and the information we're getting as a company, there were not many effective worst-case recovery and reconstitution plans in place when Katrina hit," says Chase. "A lot of plans focused on reconstituting or rebuilding at an existing location, but there were not many that called for companies to shift to a redundant infrastructure so that they would lose a minimum amount of time and revenue."
There are, to be sure, companies that got it right. Days before the storm hit, $285.2 billion Wal-Mart Stores Inc. had dozens of managers and support personnel camped out in the company's emergency command center in Bentonville, Arkansas. At the same time, Wal-Mart warehouses began sending truckloads of emergency supplies, from generators to dry ice, to staging areas where they would be available to stores that needed them. (Ultimately, 173 Wal-Mart facilities were affected. By late September, all but 12 had been reopened.) Meanwhile, privately held vacuum maker Oreck Corp. relocated its headquarters from New Orleans to an IBM Corp. disaster recovery center in Dallas, and, thanks to a backup computer system in Colorado, didn't miss a beat in issuing paychecks for its 1,200 employees. Hancock Bank in Gulfport, Mississippi, was also on the prepared list. Although it saw its 15-story headquarters ravaged by a 30-foot wall of water, it didn't lose critical data: in the days before Katrina, the company shipped its records to Chicago on 200 backup tapes.
Still, there were many employers whose plans fell tragically short — or that didn't have plans at all. Many were small businesses for which relocation wasn't an option. Others were the hospitals and nursing homes that lost patients when their emergency generators didn't work. Even companies that had recovery plans had a rough go of it. Gilsbar Inc., a $35 million insurance services firm in Covington, Louisiana, missed the eye of the hurricane by about 30 miles but still saw its hometown hammered by heavy winds. About 10 percent of its 350 employees lost their homes, and some lost family. Although Gilsbar suffered only cosmetic damage, the company was left without electricity, telephone service, or Internet access, and was forced to retreat to a SunGard backup facility in Chicago to continue serving customers.
Family Phone Numbers
If the biggest lesson of Katrina is that "worst case" can be much worse than people imagine, its other lesson is that without people and the means to communicate with them, no disaster recovery plan is worth much. In Covington, a day and a half passed before Gilsbar executives could even assess Katrina's fallout. "You'd have to have been there to understand the devastation," recalls Neal Hennegan, the company's director of technology. "You couldn't physically drive to a lot of locations. There was no gasoline. Roads were in bad shape because of trees and power lines down. Cell phones didn't work. It took a day and a half for our executive team to get together, and when we did, we met on a dark road in the woods."
The company is strengthening its disaster recovery plan to further address phone and Internet communications if it ever suffers a similar hardship again. "We've talked about satellite phones," Hennegan says with a shrug, "but I predict that people won't have them, or the networks will be down when we need them." The company has established an externally hosted employee bulletin board where workers with access to the Internet can log in, and an externally hosted toll-free employee phone line. Still, Hennegan observes, "one requires access to the Internet and the other to a phone, and a lot of people post-Katrina didn't have either."
Johnson Controls Inc., the Milwaukee-based $26.6 billion automotive systems and facility-management and control company, also found communication to be an issue. The company had about 350 employees in the hurricane's path, according to Eric Reisner, vice president of strategic programs for the controls group, including about 170 in New Orleans and about 20 who lost their homes. In the aftermath, the company had trouble finding them. "I would say our recovery planning didn't go deep enough," he says. "We had everyone's home phone and cell numbers, but not the numbers of their parents, aunts, uncles, brothers, and sisters. We didn't have any way of getting hold of our employees, and they didn't have any way of getting hold of us." Although the firm maintains a communications hotline for things like bomb threats or other localized emergencies, Reisner says, Katrina identified the need for an emergency line that is readily available in the event of a broader disaster.
Reader Comments» Post a comment