New Madrid Aftermath @ 09:25 pm
I don't usually write about my "other life" in this blog. I try to stick mostly to Fedora related things, and to save my other comments for other forums. In particular, my ham radio life is pretty unusual, and lots of what I do requires a lot of explanation. Well, this is one case I think it is worthwhile. I apologize in advance for the length.
Back when we were preparing the schedule for Fedora 15 I tried pretty hard to get someone else to take point on the Release Notes. I knew that this time of year is generally very busy for me, and that this year in particular was going to be horrible. I wasn't able to find anyone dumb enough to stand in for me, and I wasn't disappointed in how frantic it was going to get, either.
Tuesday was the peak of the madness, and it was wonderful. To understand how wonderful, though, requires quite a bit of background.
In my non-Fedora life, I am the Section Emergency Coordinator for the State of Michigan. That means that I organize about 2200 volunteers to provide backup communications in the event of an emergency. In that role I spend a lot of time with the State Police in Lansing. In Michigan, the State Police have responsibility for emergency management and homeland security. Not all states are organized this way.
When an emergency occurs, the affected jurisdiction activates an "Emergency Operations Center" or EOC. The point of the EOC is to collect representatives from all the important agencies in the jurisdiction so that the responders at the incident can get any resources they need.
When an incident exceeds the capabilities of an individual county, the State Emergency Operations Center or SEOC is activated. Every agency in Michigan government has a representative at the SEOC when it is activated. In addition to the twenty-odd government agencies, two volunteer agencies are there; the Red Cross and amateur radio (me).
Michigan is fortunate in that we don't face many of the hazards that other states have to deal with. Sure, we have tornadoes, but they tend to be localized. And we have winter storms, but they are slow moving and relatively easy to deal with. But we don't do earthquakes, tsunamis, hurricanes, lots of the things that other states deal with we simply don't face.
We do, however, have three nuclear power plants in the state, and even though the likelihood of an issue at these plants is quite low, they do represent the highest perceived threat. As a result, this is what we exercise. And we exercise nuclear power plant incidents frequently.
This really isn't all bad. A nuclear incident is one of the few incidents I can think of that requires every state agency to be involved. The complexity of this sort of incident is surprising, but every few months on average I am down in Lansing practising for a serious event at one of these plants. All of us that work in the SEOC have taken radiological training, and most of us have had to be take tests on what the plant operators would do in specific circumstances, so that we understand what is happening "in the field". (The NRC is pretty detailed about what plants have to do when certain things happen).
Many of you may have heard that this week, FEMA is conducting a "National Level Exercise" in which several states are practising for an earthquake in the central U.S. Because the geology of the area is so different, the effects of an earthquake in the New Madrid - Wabash fault area are quite a bit more devastating than west coast earthquakes. Fortunately, severe quakes in that area don't happen very often, but it turns out we are overdue. 
An earthquake of the size that hit the area in 1812 would probably not cause a lot of direct damage in Michigan, but the effects would still be devastating. You see, most of our power and petroleum runs through that area. Many major communications lines also go through the fault. An 1812 size event would knock out most of the bridges across the Mississippi, disrupting deliveries from the west coast. Many air deliveries are routed through Memphis, whose airport would be disabled for months. Huge numbers of evacuees would likely come to Michigan. All in all, the aftermath would be a big problem.
Michigan, like a number of other states, opted out of the National Level exercise, choosing instead to conduct a separate exercise around some of the impacts to our state. I was part of the team designing that exercise, which is what has chewed up a lot of my time over the past few months.
There is a lot to designing an exercise like this. FEMA has a lot of guidance on designing exercises, so the first thing on the agenda for me was a raft of FEMA courses around exercise design and evaluation. Amateurs recognized that this raised issues we hadn't considered before, so we partnered with our counterparts in Indiana, and the Indiana Department of Homeland Security to engage in some testing with that state around this sort of scenario.
The heavy lifting, though, was really done by a number of other agencies. The Public Service Commission did modelling to help understand the likely impact on electrical, gasoline and natural gas supplies. The Red Cross did a lot of work understanding how they would shelter thousands of evacuees. The Department of Agriculture went to work understanding how to deal with all the pets those evacuees would bring with them. And it goes on and on and on. The level of complexity is truly astonishing.
The way these exercises are developed is, once you have the general scenario (an 8.1 near Memphis, followed by a 6.6 aftershock in southern Indiana) you then lay out a general timeline of events. From the timeline, you then develop "injects". These are typically messages coming into the SEOC. They often are directed at the wrong agency, as they would be in a real event. As the detailed event list develops, you also need to identify how you expect the players to respond. Ultimately, you will evaluate how the players reacted compared to your expectations and plans.
We probably ended up with far too many injects. We weren't able to get them all in and even so, the players were at a dead run for the entire exercise.
As I said, we exercise nuclear power plant incidents regularly, but we haven't exercised other incidents all that often. In addition, many of the normal representatives at the SEOC were involved in the design, and thus couldn't play. Many others were needed in the "SimCell", the place the messages come from, as subject matter experts. As a result, many of the players were inexperienced in the SEOC. Even those that had worked there before were faced with issues they had never dealt with.
Although the stress level was high, it was quite impressive to see how the players reacted. There were many, many creative decisions made, and incredible cooperation between the agencies. During the exercise I was serving as an evaluator, which gave me a chance to observe much more of what was going on than I usually can.
So now that the exercise is behind us, we still have to get together and collect our evaluations, write the after-action report, and then the hardest part, the improvement plan. My counterpart in Indiana and myself have already identified a huge amount of work we need to do going forward, and we need to document that as well. But the frantic, high-stress part is largely behind us.
All in all it has been quite an experience. I can't help but feel that the citizens of Michigan are just a little bit safer than they were before, and that I made a real contribution. it was discouraging to be less engaged in this release than I would have liked, but I feel like I made a worthwhile investment of my time, and to be honest, to be quite privileged to have had the opportunity.
 There is a short video on this fault at: http://www.history.com/videos/mega-disas
 FEMA has a large number of emergency preparedness courses available online. Some are pretty specific, but there are a few that would be helpful to anyone: http://training.fema.gov/is/crslist.asp