Click to learn more about author W. Curtis Preston.

You might think that restoring the operating system and data for hundreds of computers would be your primary challenge in a disaster recovery; however, a recent podcast interview with someone who went through an actual disaster recovery says otherwise. This is the second in a series of articles about disaster recovery based on this interview. Last month’s article focused on the first challenge “Ron” experienced (as mentioned in last month’s article, we are keeping Ron and his company’s identity secret) when his company flew him to a hurricane-ravaged island: There was no internet connectivity. This month, I’m going to focus on another thing that is typically overlooked in disaster recovery planning: basic infrastructure.

In a natural disaster or major terrorist attack, basic infrastructure services are often simply unavailable. The first problem that Ron had was simply getting to the island. There were no commercial flights to speak of to get there, but luckily Ron’s company had a private jet. I’m sure he felt very important as one of the very few passengers on this flight. He landed and immediately went to work.

The good news is that there were two data centers, and one of them was relatively undamaged — “relative” being the operative word here. I’ll call this data center DC2 and the other one DC1. DC1 wasn’t so lucky and had about a foot of water in the data center. They used typical computer racks, so that meant that many of the servers escaped water damage and could be retrieved for use in DC2. So the first task was to physically rescue as many functional servers as possible from DC1 and transfer them to DC2 to begin the restore. (Ron did happen to mention that some of them got a little damaged in transit, and they had a little trouble getting them into the computer racks in DC2.)

My question is this: What if DC2 had also been damaged? Maybe DC2 was undamaged because it was chosen for its higher elevation, and that’s why it wasn’t flooded. But natural disasters and terrorist attacks often find ways around your site choice. I wonder what Ron would have done had he arrived to find two flooded data centers, neither of which had power.

Ron’s entire recovery scenario was exacerbated by the fact that he was on an island surrounded by hundreds of miles of water. There are a lot of things you can do when there are no buildings that survived a recent event — if you’re on the mainland. Most — if not all — of those options simply are unavailable if you are on an island. If your facility is on an island in Hurricane Alley, you might want to think about that.

Make sure to determine how you’re going to provide the basic infrastructure the data center would need in order to begin your disaster recovery. Where will you locate your recovery site? How will this recovery site get power? How will you get your backup data to the recovery site? There are answers to all of these questions, but they really need to be determined prior to actually initiating a disaster recovery.

As far as electricity is concerned, this can be managed with a reasonably sized generator, and most large data centers do have such generators. Ron’s recovery took two weeks. Do you have enough fuel for your generator to run for two weeks? I’m guessing not. What’s your plan for getting more fuel, and how does a disaster or terrorist attack affect that plan?

Basic Human Needs

There were no hotels or motels for Ron to sleep in, nor were there restaurants for him to eat at. And yet, he needed to do both of these things every day for two weeks. I don’t know if this was part of their disaster recovery plan, but it was somewhat amusing to hear his story about how they turned conference rooms into bedrooms. They had conference rooms that were not being used due to the disaster, and each conference room became someone’s bedroom. That means they were using the office building’s bathrooms, as well. I hope they blacked out the windows for some privacy, and I hope that they were equipped with bathing facilities. Ron did say that he never went hungry, but he did eat a lot of rice and beans for two weeks.

Large-scale disaster recoveries take a long time, and the people that are executing those recoveries need a place to eat, sleep, and bathe. Ron’s company may do with conference rooms, but he was told to bring his own bedding. If your DR plan is to use an office building as a living facility, make sure to provide basic human comfort, like a comfortable place to sleep and a place to bathe. I also think you should have emergency rations for a certain number of people.

This also means that you need running water. I didn’t hear any complaints from Ron about the lack of running water, so I’m assuming they at least had that. But your facility might not have running water still in operation. Any large hurricane or tornado can corrupt the water table and therefore the freshwater supply. How will you supply fresh water for your people to bathe in, drink, and cook with? And yes, it should be heated as well.

Plan for Worst-Case Scenarios

Think about the kinds of things that can happen to your facility and then figure out how to overcome them. How will you handle the complete loss of basic infrastructure? Where will you put your recovery data center, and how will it get power? How will your people eat, sleep, and bathe? If you plan for the worst before it happens, things will be a lot less shaky if they actually do happen.