Homeland Security Watch

News and analysis of critical issues in homeland security

April 26, 2011

A half-baked idea about measuring preparedness

Filed under: Preparedness and Response — by Christopher Bellavita on April 26, 2011

President Obama “wrote” the following as a part of PPD 8:

…I hereby direct the development of a national preparedness goal that identifies the core capabilities necessary for preparedness and a national preparedness system to guide activities that will enable the Nation to achieve the goal. The system will allow the Nation to track the progress of our ability to build and improve the capabilities necessary to prevent, protect against, mitigate the effects of, respond to, and recover from those threats that pose the greatest risk to the security of the Nation.

In previous posts over the past few years, I noted how difficult it is to “track the progress of our ability to build and improve capabilities.” At least a half dozen pilot projects have made the attempt. But tracking “preparedness” (my shorthand for this complex undertaking) still waits in the “too hard to do” box.

Since nothing is impossible to the person who does not have to do it, I think “we” (people who care about assessing preparedness) should just ask homeland security officials across the country what threats they are prepared to “prevent, protect… mitigate…” etc., and what they are not prepared for. And we should track responses annually.

However I detect serious misgivings from people-in-charge-of-things about the responses they would get from that kind of inquiry. There seems to be an expectation that homeland security officials charged with taking care of their jurisdictions would lie about their preparedness so they get more money.

Or maybe even worse than lying, their responses would be “subjective.”

So just asking responsible professionals how prepared they are is a naive non-starter, unless the request can be backed up by some kind of six sigma truth guarantor.

What else can one do?


This gets to the topic today’s post: Where do new ideas come from?

In my experience they come in intermittent, varied and interrupted intensity, like bees getting called to pollen

Truly blessed is the man or woman who gets to work with people around whom they can think out loud. I have the good fortune of working with people like that, who do not expect your ideas to be fully formed.

Here is an exchange I had with three colleagues (Ted Lewis, Rudy Darken, and Lauren Fernandez) last week about another idea for assessing preparedness. The idea is “half baked,” by which I mean in a non-derisive way, it is not done.


Me: When you get the chance, can you look at this 6 minute video from David Snowden, the complexity theorist. [The video is posted in this blog, below.] It is a promotional video for SenseMaker software, but it describes a kind of narrative research that could be used by states and cities to assess preparedness.  It basically uses stories as data, and then portrays those data as a (both fixed and dynamic) fitness landscape.

I wonder if we could use an approach like this to measure (or at least portray) preparedness.


Ted: While you’re at it, take a look at this video (from the United Kingdom, about involving people in their energy future).

It might be a model for doing what you want.


Rudy — Conceptually, I get what they are doing, but I can’t quite grasp what form this takes or how I would use it [to measure preparedness]. Do you?


Ted : What is the objective? What do you want to achieve? Then work backwards. If you were going to construct a Maslow Hiearchy for homeland security preparedness, what would it be? That is the goal. Once you have that, what data do you need to collect and display to define or ‘prove’ a layer in this hierarchy? At each layer, drill down to the precepts. What are they.  Keep going until you are exhausted or the listener is no longer interested. Then you are done.


Rudy — The problem is that the connection between things we do and preparedness outcomes is fuzzy at best. In fact we don’t even know what the metrics are. So that’s what I’d want to use the wisdom of the crowds for. If we could think of a way to get hundreds of responses that help frame the concepts of what goes into preparedness and how much does it matter, you might have a very interesting result.


Ted — agree! What are the metrics? It could look something like the UK energy site, but what are the levers?


Rudy — Exactly my point. What if you had a way for the user to say what he thinks the levers are and how he thinks the levers influence outcomes (which he would also have to specify, or pick from a list).


Me — Unless I’m missing something, the UK app illustrates trade off analysis.  The goal is clear and measurable (achieve 20% of 1990 CO emissions). I think preparedness assessment could lead to trade off analysis, but I think there are some steps missing before we get there — like what the goal is.

As I understand a fitness landscape, it is an image that portrays the relationship between agents and their environment.  Some of those relationships are stable and are unlikely to change without a massive disruption.  So one might say those agents are a good fit with their environment.  Other relationships are unstable and are not likely to persist in the environment (such as paying attention to terrorism in a community that has never had a terrorism incident).  So preparedness would mean something different in each community (or more accurately, each region).  By first mapping the preparedness landscape (by collecting and plotting stories told by agents as described in the SenseMaker software), one would have a baseline from which to look at how preparedness is changing.  This would be analogous to observing how an ecosystem changes as its environment or agents within the system change.

Public choice theory posits that, over time, communities tend to be satisfied with the level of government services they receive.  If they are not, they will tend to elect people who will work to change the level of services — via ending, starting programs; increasing/decreasing taxes.  There’s more to it than that, but public choice assumes a tendency toward a dynamic equilibrium.  I wonder if one can make the same assumption about preparedness: that communities are generally satisfied with their level of homeland security preparedness, otherwise they would take action to change that level.  (This excludes for now one level of a system expecting more or less out of a different level of a system: fusion centers might be an example of that.)

The key to all this, as I understand it, is in one of Ted’s earlier emails: define the data.  I think the objective is agent satisfaction.  The data consist of stories the agents tell about prevention, mitigation, response, recovery, resilience, and the other portmanteau concepts in preparedness – and I think identifying the questions that elicit those stories would be a useful next step.

I want to bring Lauren Fernandez into this conversation.  While she was with DHS, she led pilot projects to figure out how to assess preparedness (in response to HSPD 8 requirements).  She has more experience and knowledge about measuring preparedness than anyone I know.  Here’s what she wrote in an email to me yesterday on this topic:


Lauren: I watched Snowden’s video, and some of his statements resonated with me—in the real world people share anecdotes, and this information is valuable. I liked the idea of the storyteller also being the interpreter. I didn’t quite understand what a fitness landscape is and how it can be developed.

I do think that different ways of assessing preparedness need to be explored. It is the type of problem where there are not (nor do I think there can be) commonly accepted and measurable indicators across the board. There are some instances where quantitative measures make sense (e.g. average response time of certain equipment in certain geographic area), but this can’t be done for everything. There are instances when people’s perception of performance is adequate, and instances when it is not.

There are so many moving pieces:

– Preparedness requires dollars, and there are differing views on where dollars should be spent and how much the government should be able to acquire and distribute for all needs (education, healthcare, security, etc.) [Bellavita note: this would be a good use of an application similar to the UK energy app.]

– The actors that wish to cause harm in the domestic US are not static, they adapt. Our defenses must also adapt, making measurement difficult.

– Our capabilities to prevent and respond change over time (new technologies, events affect our resources, etc.)

– Some things are just difficult to measure. If you provide enough specification in a question to get a decent answer, that answer is not generalizable. If the question is too generalized, it can’t be answered well.

– We measure because we want to adjust where we place resources (time, money, etc.) People who are a part of the system being measured are motivated to affect the evaluation (it affects programs they care about, personal finances, reputations, etc.). Measuring teacher performance is a good analogy—we end up placing a lot of resources into standardized testing. Teachers spend significant time working to get students to pass the tests. To what end? Has this produced better students that will improve their personal and community life?

When thinking about a new approach to preparedness measurement, perhaps we can start with the end in mind [Bellavita note: to me, the desired end in mind is regional satisfaction with that region’s preparedness for certain types of threats] and work backwards to tools that will help us meet that end.

I liked this blog post on the importance of being able to take action. I’m curious if there is any evidence that fitness landscapes can (or could) influence decisionmakers and if the information developed is good enough to make decisions.


Ted — A couple of observations: You may be describing qualitative reasoning, which is a subtopic of artificial intelligence. No numbers, just directions like up, down, small, big, etc.  Alternatively, your model might simply be “the wisdom of crowds”, and nothing more than a heat map of the US. Ask people to rank their level of satisfaction along a scale.  At least Chris identified the goal: to reach preparedness satisfaction. That is a start. What stories deal with satisfaction?


Me —  If asked, I could come up with stories about how satisfied I am with my community’s ability to anticipate, respond to and recover from floods.  I could do the same with fires and wind storms.  Those are the major threats where I live, and there are stories people in the community tell each other about, for instance, the wind storm of 1995 that destroyed a lot of trees, knocked out power for a week or so in some places, disrupted travel and so on.

Without actually collecting those stories (yet), I am guessing I could locate on several triangles (like the Snowden video) how prepared I thought my community was to anticipate, prevent, mitigate, communicate about (and so on) a major windstorm. That could provide some measure of satisfaction — or at least a basis for asking. I could also include indicators of likelihood, vulnerability, etc.  Same things with stories about fires, hazardous materials (Interstate 5 runs right through the town), and terrorism  — we have a machine gun store in town (under the right conditions you can rent a machine gun). It’s a few blocks from an elementary school.  There’s a story.

Using an app in Walmart, on iPhones, in schools, in the library, all over the place, people could tell and interpret their preparedness stories; based on the “data points” (still not sure what that means) we would get a fitness landscape.  What that landscape means – and what its action implications are – is also unclear to me.

But the task is to collect and display stories.  Then let’s see what emerges.


Ted — And would you open this up to the crowd? You still need some way to rate or compare “stories”.


Me – I would open it up to the crowd — i.e., US residents — the people who are supposed to benefit from all of this preparedness.  As I understand the SenseMaker model Snowden uses, the designers create the categories to be rated. The story tellers are the ones who rate their own stories.  The video describes the process, from the 1:13 mark to the 4:15 mark.

And to emphasize again, I’m stumbling around here.  If preparedness is a complex idea and activity, maybe a tool sensitive to complexity could help give a picture of preparedness (I hesitate to say “measure preparedness”) in a way that would allow policymakers/analysts to try to explain what accounts for the different preparedness landscapes and whether a particular landscape is politically, fiscally, socially acceptable….  (and here I trail off into mumbling).


Lauren — There are a number of studies that focus on trying to evaluate the preparedness of citizens.  (DHS tracks this research here: http://www.citizencorps.gov/ready/research.shtm).  Although there are ‘rate your community’ surveys for general attributes, I am not aware of studies that focus on citizens evaluating the preparedness of their communities.

Some questions to consider (meant not to attack this idea, but rather to strengthen it through critical thinking):

– Do citizens know enough to develop a good picture?  (Perhaps ‘enough’ is the wrong question, as different individuals will have different pictures which in aggregate is valuable, but do they have key insight.)

– Do citizens know how their emergency services prepare?  In my mind, knowing that the fire and police chiefs talk over pancakes once a month (as they apparently do in Austin, Texas) would make me feel more prepared than having a spiffy new EOC.  I know we have a spiffy new EOC in [my city], but I don’t know what the relationship between fire and police is like.

– What would and could policymakers do with this information [about preparedness]?  (Would they shut down the machine gun rentals near the elementary school?)

– I think Rudy’s statement that the “connection between things we do and preparedness outcomes is fuzzy at best”  is very true.   We have pulled multiple levers in the past 10+ years (fusion centers, WMD response equipment, Citizen Corps, etc.) but don’t have a good handle on the delta of preparedness.   Is there a good way to determine the relationship to outcome, or are the events we are preparing for so rare and complex that we can’t do this well?

– That said, there may be a few exceptions in understanding the connection of actions to preparedness. We’ve learned that people don’t want to evacuate without their pets.  Seems trivial, but by satisfying this behavioral factor by planning to shelter pets, I think (and perhaps there is evidence) that preparedness is improved.  What other behavioral keys exist?  What makes people feel comfortable reporting suspicious behavior?  Could a good outcome of a study be to encourage more study on the impact of levers first before developing standards?

–  What would and could citizens do with this information?

–  Are citizens rational actors?   Recent developments in behavior economics have found that our assumption that investors are ‘rational actors’ is not a good one.  I wonder if preparedness satisfaction might be very tied to psychological factors.  Or would public choice theory ring true?

– Should communities be preparing for something greater than what they have experienced?  Stories from the Midwest may relay a high level of satisfaction, but the New Madrid fault hasn’t been shaking.

Mumble, mumble…

Ted —  Most of your questions boil down to the question, “do you trust the wisdom of crowds”?

Is perception about preparedness reality? Probably.

Fukushima illustrates this. Nuclear power is just as safe ( or not) now as it was before the earthquake, but I bet the landscape of preparedness satisfaction is radically different now than before!


Rudy – This is fairly well outside my area of expertise, but from what I’ve seen, true preparedness is unachievable without large-scale community (i.e. private citizen) involvement. That implies some level of ownership of the problem. There currently is little to none of that. It would be interesting to ask a large swath of the population how likely they think their neighbors would be to come to their aid if they needed it — like when a tornado rips through a neighborhood, damaging some homes but not others. Then how likely would they be to aid their neighbors. Then broaden to include the role of responders. I wonder if the fitness landscape for preparedness is a good idea but one that needs to be built in incremental layers, not all at once.

Ted — Is it possible to build a landscape out of the words people use to tell their stories? What if nobody comes to our field of dreams?




Share and Enjoy:
  • Digg
  • Reddit
  • Facebook
  • Yahoo! Buzz
  • Google Bookmarks
  • email
  • Print
  • LinkedIn


Comment by Dave Snowden

April 26, 2011 @ 4:39 am

A brief response. We have just completed a SenseMaker project to measure the resilience of a country to sudden or unexpected change, and are starting one that looks at finding the space for alternative (not counter) narratives to drug barons. All of those use the landscapes to identify and measure the dispositions of what is a complex system to identify possible directions that can be influenced at a micro level in order to evolve towards a better future, and/or create resilience capability.
We can use wisdom of grounds (I prefer to call it distributed cognition) but its critical to design it, crowd sourcing is not the same thing – the stupidity of herds.
In Washington DC 26th/27th April (ie now) if anyone wants to talk. Also teaching some of this stuff on public courses in San Francisco/San Jose and Vancouver in a few days time.

Comment by Philip J. Palin

April 26, 2011 @ 5:22 am

Ted’s points about crowds and Rudy’s about participation strike me as especially important.

If the conversation above had:

1) included a wider range of participants and
2) focused on the resilience of a particular place or sector or “set”…

Then,especially with reasonable facilitation, some formal and informal preparedness decisions would have emerged. These might well include contending decisions among sub-groups. Consensus can be helpful, but is not required. Decisions may, often will, include goals, but not always.

As a result of these decisions having emerged from active participation, there is plenty of evidence from a variety of disciplines that action — and passive support for the action of others — will result.

It can then be observed and described how the groups proceed to collaborate, deliberate, and implement their decisions (or not). Differences in efficiency, effectiveness, and other measures can be discerned and reported out.

It is the reporting out and the hearing of the reports — the story-making — that may have the most potential for prompting sustained and meaningful participation, collaboration, deliberation and action regarding preparedness. The better the story, the more impact on preparedness… even (especially?) if the story is one of failure. (See Aristotle, Poetics)

Comment by William R. Cumming

April 26, 2011 @ 7:30 am

Great post and comments. The TCL date from 2007 and I understand now under revision. Anyone have history on who or whom prepared the 2007 TCL and who is now working the revision?

By the way note that the National Response Framework contemplated that a number of plans and response systems would be developed (that NRF issued January 2008) and progress reports available?

I have gone back and looked over Congressional oversight of Executive Branch capabilities and have concluded there has been NO oversight! Anyone agree or disagree?

Are their any good links to articles or discussion in publications or books of the history, implementation, etc. of the TCL?

Do you think the authors of PPD-8 understood the current TCL and did they contemplate major changes and if so why or why not?

A fool of course can ask more questions than wise men and women can answer?

Comment by Donald Quixote

April 26, 2011 @ 8:04 pm

The homeland security officials shall likely report that they are almost prepared for what has already happened – if you can continue the funding. Would the tracking of their responses assist in the evaluation process or be used against them when the level of preparedness claimed was disproven by their response to the next major disaster or incident?

By the way, what is preparedness? Or even more important, what is homeland security? The cynic could say that whatever they may be, they sure pay well.

Pingback by Homeland Security Watch » Innovating with Integrity

April 27, 2011 @ 12:02 am

[…] Chris Bellavita let us in on a discussion he’s been having with a couple of his colleagues about the power of narrative as a means of […]

Pingback by Innovating with Integrity « R4 Resilience

April 27, 2011 @ 2:19 am

[…] Chris Bellavita let us in on a discussion he’s been having with a couple of his colleagues about the power of narrative as a means of […]

Pingback by Halloween costume idea for a couple? | Neytiri Avatar Adult Costume.com

June 10, 2011 @ 1:56 pm

[…] Homeland Security Watch » A half-baked idea about measuring … […]

RSS feed for comments on this post. TrackBack URI

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>