It’s been a challenging Atlantic hurricane season this year, to put it mildly, with the likes of Hurricane Harvey and Irma inflicting horrific damage in southern Texas, Florida and the Caribbean. Also, Mexico has experienced two major earthquakes in the span of two weeks. Our thoughts go out to those impacted and we certainly hope those in the afflicted areas have been able to weather the disasters safely.
In my role at APC by Schneider Electric, I understand how critical it is to prepare for and respond to natural disasters like hurricanes and earthquakes. We partner with customers around the world to supply power and cooling infrastructure IT solutions and understand the importance of getting back online as quickly as possible after a natural disaster.
It’s a topic we’ve covered previously in a number of blog posts, such as this one describing the elements of an emergency plan and this one with some last-minute tips for hurricane preparedness. We’ve also covered the business impact of downtime caused by power outage in this blog which makes the case for a strong business continuity plan.
But preparedness isn’t just about taking steps to prepare for an event; it also involves a thoughtful process for recovering after the fact. With Harvey and Irma now in the rear-view mirror, and Maria still lurking as of this writing, in this post we’ll look at steps to get IT equipment back in action following a major natural disaster.
In the aftermath of a hurricane or earthquake, it’s likely you’ll be facing power outages, and perhaps Internet Service Provider outages. You may also face structural damage to roofing, roof-top chillers, windows and the like.
Given that, the most crucial issue to attend to is simply the safety and availability of site personnel. You need to ensure the availability of safe shelter for onsite personnel, along with food and water. You can’t get anything done without personnel who have a safe place to work.
In some cases, you may have IT folks who have been on-site for the duration of the event. If so, they will likely be tired and ready to go home, so you need to ensure the availability of additional staff to replace them. You’ll need to determine whether replacement staff will be able to get to wherever your IT gear is, or whether they’re subject to flooding, blocked roads or other issues. Be prepared to recruit staff from other sites to relieve staff who were on site during the event if necessary.
Safety is paramount in the aftermath of any disaster. With respect to hurricanes and their potential to bring flooding, the safety of your power infrastructure is critical. Assuming you have lost power, the first order of business when assessing your power situation is to check for any issues with the critical path for your electrical power and correct them to ensure safety and site continuity.
Verify what the storm’s impact on the overall surrounding area is, to determine what your options are if you can’t get fuel for your generators, for example. Get in touch with the utility company to understand a reasonable time when it will be safe to transfer off generator back to the utility.
Perform preventive maintenance on all critical equipment to search for any hidden issues, especially equipment that was exposed to the elements. When power permits, check on oil and filters in your generators and replace if necessary (depending on how many hours they were running).
Conduct frequent site walk-throughs in the aftermath of the event to increase the likelihood that you’ll find problems with various equipment and infrastructure. Keep management informed of any issues you discover after verifying all areas and systems.
As time permits, immediately start collecting “lessons learned” and information on any “near miss” situations while events are fresh in everyone’s minds. Document any areas that could be improved during the next event.
Finally, immediately following the event, start coordinating and scheduling any required break fixes and corrective actions for IT gear and other infrastructure.
This is by no means an exhaustive list of post-hurricane or disaster recovery tasks, but it’ll get you headed in the right direction. If you’re lucky enough to be reading this without having recently suffered through a natural disaster, hopefully it’ll prompt you to put together a more thorough disaster preparedness and recovery plan. One place to start is our Power Outage Resources page, which has information on power protection strategies, relevant white papers and lots more. Most importantly, remember that the proper planning, experience and equipment is the most sure-fired way to ensure Certainty in a Connected World.