Somewhere, there is a hurricane brewing. It may well be heading your way. Or, say a California news station just reported that the "Big One" will definitely happen next week. If neither of those scenarios set your heart racing, think of something simpler, like the upper part of your state is flooded and you know all that water has only one direction to flow downhill to your city.
In other words, a natural disaster is about to occur.
Now let's assume, for the moment, anyway, that you haven't exactly been diligent in preparing for a disaster. After all, it couldn't ever happen to you. Except now it's about to. Or at least it might. What can you do to save data and avoid downtime without having to spend a fortune?
Lucky for you, here are a few tips to guide you through, until you put a long-term disaster recovery (DR) plan in place that will help you survive through nearly any type of catastrophe. These steps are meant to serve as a tourniquet, and we all know what happens when a tourniquet is left on too long.
Tape backups are a necessity. But they can't be relied on for rapid restores, and they may not be accessible immediately following a disaster.
"You can't use tapes for recovery, as it takes too long," says Gartner Analyst Donna Scott. "It's much more efficient to replicate data between two facilities."
That's exactly what law firm Chaffe McCall LLP, decided to do after Hurricane Katrina exposed the weaknesses of its DR planning.
"We lost access to our main New Orleans office for several weeks," says James Zeller, senior network manager of Chaffe McCall. "We restored critical functions within about a week using our Baton Rouge office."
However, that remote office lacked proper hardware to support a full recovery. Backup tapes had to be manually taken to high ground to avoid data loss. Several servers were moved from New Orleans to Baton Rouge right after the storm. Perhaps the biggest problem, however, was that the temporary headquarters didn't have enough desktops to cope with an influx of new users, so more had to be acquired.
To avoid all that heavy lifting in the event of another storm, the law firm added WANSyncHA software from Waltham, Mass. XOsoft (which CA recently acquired). This software-based approach to replication enables one office to fail over to the other. That, in turn, enables the company to remain up and running in the face of a natural disaster, without having to spend a fortune on hardware replication.
"We are now using XOSOFT for our SQL, Exchange and file servers to be replicated to standby equipment in the Baton Rouge office," says Zeller.
A good stand-by power supply is another staple of DR preparedness. Hurricanes tear up power lines, which then take a while to be rebuilt. Some companies tend to rely on batteries for such a contingency. Upgrading them is a smart thing to do in the face of trouble. It is a purchase unlikely to break the budget, but one that may not be enough if the devastation is severe.
Take the case of the University of New Orleans (UNO). Prior to Katrina, it shut down its data centers completely.
"We shut everything down, as we didn't want to rely on UPS [uninterrupted power supply] when there was no one here to monitor it," says Jim Burgard, UNO's Assistant Vice Chancellor for University Computing and Communications. "Other universities suffered significant damage after their UPS batteries went dead, and then the power came on and went off many times."
This summer, UNO upgraded its UPS and is adding a gas-fired generator, so it has an alternate power supply on campus. When completed, UNO will be able to run its servers, SAN and AC, despite city power being down.
"To remain operational, you'll need a generator and plenty of petrol/diesel which will have to be changed every six months, or you'll find that it doesn't actually work when you need it," says Clive Longbottom, an analyst with U.K.-based Quocirca.
Depending on the size and resources of the operation, however, setting up a replica site and backup power can be budget breakers.
"This sort of DR is expensive, and can usually only be done by larger companies," says Longbottom, "This, is where hosting can help."
Hosting companies have better DR plans than the majority of companies, have all the back up power that they need, tend to have multiple sites that can mirror each other, and have facilities built to withstand just about anything.
"Find the right provider that can help with the planning, personnel, and technology needed to protect your business from a disaster," says Lenny Monsour, director of product management at SunGard Availability Services in Wayne, Penn. "This is the first step in lowering disaster recovery costs."
SunGard offers a wide range of hosted options, from a mirror data center on hot standby to space to host a few servers in case of disaster. If the former can't be afforded, the latter provides a means of safeguarding at least the most crucial systems.
"It is practical to segregate your technology systems based on their importance to the operation of your business," says Monsour. "Applications that are critical to service delivery should be treated with a higher priority than back-office systems."
However, Gartner's Scott cautions that companies should not wait until a disaster is imminent before calling suppliers like SunGard.
"You need such resources to be involved in testing and recovery and your plan, so you can't just call at the time of a disaster," says Scott. "I would recommend hiring a part time consultant who can sit in on weekly meetings and become involved in regular testing of the DR plan."
Although technology is vital in an event, expert after expert stresses the primary importance of people. Even with replication in place, for example, Chaffe McCall realized it needed to do much more to be safe, particularly in the area of employee communication.
"We have taken steps to try to ensure better communication between employees after evacuation by setting up an emergency Web site and alternate Web-based email system for post-disaster communications, separate from our Exchange server," says Zeller.
The people side, of course, is generally less expensive than the technology elements that make up a comprehensive DR plan. Testing, for example, costs next to nothing yet is invaluable in the face of adversity.
"Make sure your people know what to do," says Mike Karp, an analyst with Enterprise Management Associates, based in Boulder, Color. "That means testing of your emergency systems now."
Chip Nickolett, a consultant at Comprehensive Solutions, a systems integrator based in Brookfield, Wisc., agrees. When disaster looms, he says, the first consideration should be the amount of time needed to safely get employees away from imminent danger. In lieu of a real plan, he suggests companies do as much as they can in the interim to bring systems to a consistent and known state, make backups, move away from the danger, and hope the losses are minimal. One way to do this is to work backward from the anticipated time of the event and do as much as is realistically possible within that limited window of opportunity.
"Knowing how long operations and procedures typically take is helpful data when determining what can be done in that remaining time," says Nickolett. "Getting a clean shutdown and full backups of your system, and then immediately removing those backups to a safe location is recommended."
What kind of backups? Karp suggests non-magnetic media, such as DVDs, because of their inherent durability.
"When the storm is this side of the horizon, backup the most critical data to some DVDs, put them in your briefcase and head for the door," he says. "And above all, never put yourself in the place of the IT guy who called up his vendor for support, asking 'How deep does the water have to be in the computer room before I push the red button?'"