McClatchy’s DC bureau published a very interesting piece on Sunday that tells the inside story of the Nationwide Plan Review, citing numerous examples of confusion and disagreement resulting from DHS’s survey and final results:
The results of the $5 million attempt by DHS to determine whether states and cities have adequate emergency plans are troubling, and they offer a glimpse of serious behind-the-scenes disagreements among government officials over how prepared the nation is for another catastrophe.
DHS challenged the self-assessments of almost 60 percent of the states, cities and U.S. territories that participated in the review, according to the internal records obtained by McClatchy Newspapers.
The documents, which Homeland Security officials refuse to make public, reveal that, in most of those cases, the federal agency gave the states and cities significantly higher marks than they gave themselves.
The differing assessments reflect continuing confusion over how states and cities need to handle a catastrophe and what types of disasters they should be prepared to handle. That, in turn, threatens to undermine the federal government’s $18 billion effort to help them prepare for disasters and terrorist attacks.
“There’s a substantial gap between our understanding of preparedness and that of Homeland Security’s,” said Jim Mullen, the director of Washington state’s emergency management division, which rated itself less prepared than federal officials did. “There is not a recognition of the full thrust of what has to be accomplished at the state and local level, and that’s because there’s a fundamental misunderstanding of what we do and what we need.”
….Of the 131 jurisdictions that DHS reviewed, only two – Florida and California – agreed with the federal government’s assessments. Fifty-eight states and cities rated themselves less prepared than Homeland Security did in at least four of nine categories, including evacuation plans, disaster warning systems and the ability to provide emergency medical treatment. Eighteen said they were better prepared.
The story goes on to provide several specific examples of these disagreements between state officials and DHS.
After reading this story, and looking at the Nationwide Plan Review documents, I get the sense that this process has fallen prey to risks that face any attempt to understand the world via a survey-based methodology. The capabilities, institutions, processes, and threats that compose and interact with national emergency preparedness system do not easily lend themselves to the imposition of a clear, logical template, and as a result, any effort to survey baseline preparedness is going to face a trade-off between getting results that can be meaningfully compared across states and results that acknowledge this complexity and are sufficiently rich in their findings. It’s certainly a worthwhile exercise to try to carry out such baseline assessments, but there needs always to be an understanding of the tradeoffs and imperfections in the information-collection process.