Risk Management

Rethinking the Worst Case

How disaster recovery is changing in the wake of Katrina and Rita.
Randy MyersNovember 10, 2005

To understand how Hurricane Katrina is changing the way Corporate America thinks about disaster recovery, look no further than the computer-services company in New Orleans that located its data center to the third floor of its building to preclude any possible risk of flood damage. “I salute them for planning,” says Hank Chase, director of homeland-security programs for management consulting firm Smart and Associates LLP, in Devon, Pennsylvania. “They knew they were in an area subject to flooding.” Unfortunately, he says, no one imagined the building could be inundated by floodwater 23 feet deep, which is what Katrina delivered when it broke through the city’s levees in August. The company’s data center was destroyed. “They had no backup systems, no redundant infrastructure whatsoever,” observes Chase. “The last I heard, they’re down completely — out of business temporarily.”

Of all the lessons learned from Hurricane Katrina, the most important is this: companies must rethink the definition of “worst case.” “Based on public information and the information we’re getting as a company, there were not many effective worst-case recovery and reconstitution plans in place when Katrina hit,” says Chase. “A lot of plans focused on reconstituting or rebuilding at an existing location, but there were not many that called for companies to shift to a redundant infrastructure so that they would lose a minimum amount of time and revenue.”

There are, to be sure, companies that got it right. Days before the storm hit, $285.2 billion Wal-Mart Stores Inc. had dozens of managers and support personnel camped out in the company’s emergency command center in Bentonville, Arkansas. At the same time, Wal-Mart warehouses began sending truckloads of emergency supplies, from generators to dry ice, to staging areas where they would be available to stores that needed them. (Ultimately, 173 Wal-Mart facilities were affected. By late September, all but 12 had been reopened.) Meanwhile, privately held vacuum maker Oreck Corp. relocated its headquarters from New Orleans to an IBM Corp. disaster recovery center in Dallas, and, thanks to a backup computer system in Colorado, didn’t miss a beat in issuing paychecks for its 1,200 employees. Hancock Bank in Gulfport, Mississippi, was also on the prepared list. Although it saw its 15-story headquarters ravaged by a 30-foot wall of water, it didn’t lose critical data: in the days before Katrina, the company shipped its records to Chicago on 200 backup tapes.

Still, there were many employers whose plans fell tragically short — or that didn’t have plans at all. Many were small businesses for which relocation wasn’t an option. Others were the hospitals and nursing homes that lost patients when their emergency generators didn’t work. Even companies that had recovery plans had a rough go of it. Gilsbar Inc., a $35 million insurance services firm in Covington, Louisiana, missed the eye of the hurricane by about 30 miles but still saw its hometown hammered by heavy winds. About 10 percent of its 350 employees lost their homes, and some lost family. Although Gilsbar suffered only cosmetic damage, the company was left without electricity, telephone service, or Internet access, and was forced to retreat to a SunGard backup facility in Chicago to continue serving customers.

Family Phone Numbers

If the biggest lesson of Katrina is that “worst case” can be much worse than people imagine, its other lesson is that without people and the means to communicate with them, no disaster recovery plan is worth much. In Covington, a day and a half passed before Gilsbar executives could even assess Katrina’s fallout. “You’d have to have been there to understand the devastation,” recalls Neal Hennegan, the company’s director of technology. “You couldn’t physically drive to a lot of locations. There was no gasoline. Roads were in bad shape because of trees and power lines down. Cell phones didn’t work. It took a day and a half for our executive team to get together, and when we did, we met on a dark road in the woods.”

The company is strengthening its disaster recovery plan to further address phone and Internet communications if it ever suffers a similar hardship again. “We’ve talked about satellite phones,” Hennegan says with a shrug, “but I predict that people won’t have them, or the networks will be down when we need them.” The company has established an externally hosted employee bulletin board where workers with access to the Internet can log in, and an externally hosted toll-free employee phone line. Still, Hennegan observes, “one requires access to the Internet and the other to a phone, and a lot of people post-Katrina didn’t have either.”

Johnson Controls Inc., the Milwaukee-based $26.6 billion automotive systems and facility-management and control company, also found communication to be an issue. The company had about 350 employees in the hurricane’s path, according to Eric Reisner, vice president of strategic programs for the controls group, including about 170 in New Orleans and about 20 who lost their homes. In the aftermath, the company had trouble finding them. “I would say our recovery planning didn’t go deep enough,” he says. “We had everyone’s home phone and cell numbers, but not the numbers of their parents, aunts, uncles, brothers, and sisters. We didn’t have any way of getting hold of our employees, and they didn’t have any way of getting hold of us.” Although the firm maintains a communications hotline for things like bomb threats or other localized emergencies, Reisner says, Katrina identified the need for an emergency line that is readily available in the event of a broader disaster.

By the time Hurricane Rita was en route to the Gulf Coast, says Reisner, the company had collected family phone numbers for employees in the path of the storm. “With Katrina, it took about a week to locate every one of our employees,” he says. “With Rita, it was done within 24 hours. That’s when we knew everyone was OK. And within another 24 hours, we knew what they needed, whether it was food, cash, or housing.”

Even Wal-Mart, insulated from total disaster by its size and global reach, learned a few things about communication. For example, the company routinely maintains a toll-free emergency phone line for employees. But when Katrina struck, the company brought in as many as 100 additional people to staff that line. It repeated the process a few weeks later for Hurricane Rita, and Jason Jackson, director of business continuity, says the company was again able to refine the operation. “We were able to tell associates who called in where they would be able to get their paycheck, where they could find shelter, and what number to use to [contact] the American Red Cross,” he says, adding that during past disasters, the company would simply give that information to managers to share with individual employees.

Having communication cascade from the top also proved invaluable during Katrina, with Wal-Mart CEO H. Lee Scott Jr. participating in twice-daily conference calls with his direct reports, their direct reports, and other key members of the disaster recovery team. “This was new for us,” says Jackson. “Before, we would just have a lot of briefings and individual communications from the emergency-operations center, and a situation report that would come out once or twice a day.” Not only was senior management better informed under the new approach, he says, but problems got prompt attention. “It helped us organize and resolve issues quickly.”

Correcting Errors

Figuring out what does and doesn’t work is an art at Wal-Mart. Even as he was scrambling to help the world’s largest retailer recover from Katrina, Jackson was making notes about its disaster recovery efforts. “Everyone was keeping their own little COE sheet,” he recalls. COE, in Wal-Mart parlance, means “correction of errors.” Jackson was coming up with so many observations that he bought a digital recorder to log them verbally.

It also pays to learn from your mistakes. When a series of four hurricanes hit Florida in 2004, Wal-Mart discovered it could better answer questions from employees about supplies, shelter, and other services if it had a member of the American Red Cross in its emergency command center; for Katrina, it did. This year, after working through Katrina, it decided before Rita’s arrival to ratchet up its contact with emergency-response personnel another notch by stationing two of its own managers with the Governor’s Emergency Management Division of the Texas Department of Public Safety, where they were able to work directly with government disaster-response personnel.

Other companies learned not to depend on government agencies to meet their needs. Instead of waiting for the Federal Emergency Management Agency or other organizations, Johnson Controls sent 20 recreational vehicles to Amite, Louisiana, after Katrina to serve as temporary housing for employees. It also set up an intranet site to match employees in need with goods donated by their fellow workers. Both measures will become part of future disaster planning, says Reisner.

Meeting such basic needs, says Johnson Controls spokesman Darryll Fortune, is more than just a moral issue. Reisner agrees. “If you have employees who are unsure of their future, it doesn’t help anyone’s productivity,” he says, noting that Johnson Controls had 90 percent of its workers who were affected by Katrina back on the job within two weeks, and 100 percent within three weeks.

More Redundancy

Whether Katrina and Rita prompt companies to plan adequately for the next catastrophe remains to be seen. “What we have found after virtually every major disaster is that people plan for that one to happen again,” says David Palermo, vice president of marketing for SunGard Availability Services. “After 9/11, companies built backup centers that were 30, 50, 100 miles away, thinking they would be far enough out of Manhattan. Then the Northeast power outage came, and that covered 11 states. So people started building backup power supplies. Now the hurricanes have shown us another kind of destruction. Who knows what the next disaster will be?”

Whatever it is, Gilsbar’s Hennegan predicts that many companies will tweak their plans to incorporate “a lot more redundancy, both in terms of geography and suppliers.” Since Katrina, Gilsbar has inked contracts with outside vendors to create a mirrored Website that would be switched on immediately if its own site gets knocked out, and to take over the company’s inbound toll-free calls until employees can get to a backup SunGard site.

Too much? After Katrina, that would be a hard argument to make. After all, says Anna Winningham, vice president of operations with Risk Control Strategies in New York, “It may be human nature to believe ‘it won’t happen to me,’ but nature continues to prove humans wrong.”

Randy Myers is a contributing editor to CFO.

After the Storm

First-mover advantages don’t just accrue to companies that are first to enter a new market; they also accrue to firms that are first to begin rebuilding after a disaster. “Construction materials and resources become exceedingly tight after a disaster,” warns Ray Dufresne, vice president of consulting services for VFA Inc., a facilities-management consulting firm. “If you’re behind the curve, you’re not going to get your work done.”

One way to stay ahead of the curve is to make sure you have a readily accessible inventory of your company’s buildings and contents, from people to PCs to phones and software systems. Knowing exactly what you have and what’s been lost lets you be among the first in line to hire contractors and get started on rebuilding.

Wal-Mart Stores Inc. maintains an intranet application, called the Wal-Mart Incident Management Website, which provides a near-real-time summary of what is going on at each facility. The Website allows users to drill down to learn about the status of systems at each store, any physical damage, and issues affecting employees. In the midst of Katrina, Wal-Mart IT staffers created several on-the-fly enhancements, including linking the systems with geographic information systems software that allowed the company to create maps showing which facilities were facing which kinds of problems. — R.M.