Homeland Security Watch

President Obama “wrote” the following as a part of PPD 8:

…I hereby direct the development of a national preparedness goal that identifies the core capabilities necessary for preparedness and a national preparedness system to guide activities that will enable the Nation to achieve the goal. The system will allow the Nation to track the progress of our ability to build and improve the capabilities necessary to prevent, protect against, mitigate the effects of, respond to, and recover from those threats that pose the greatest risk to the security of the Nation.

In previous posts over the past few years, I noted how difficult it is to “track the progress of our ability to build and improve capabilities.” At least a half dozen pilot projects have made the attempt. But tracking “preparedness” (my shorthand for this complex undertaking) still waits in the “too hard to do” box.

Since nothing is impossible to the person who does not have to do it, I think “we” (people who care about assessing preparedness) should just ask homeland security officials across the country what threats they are prepared to “prevent, protect… mitigate…” etc., and what they are not prepared for. And we should track responses annually.

However I detect serious misgivings from people-in-charge-of-things about the responses they would get from that kind of inquiry. There seems to be an expectation that homeland security officials charged with taking care of their jurisdictions would lie about their preparedness so they get more money.

Or maybe even worse than lying, their responses would be “subjective.”

So just asking responsible professionals how prepared they are is a naive non-starter, unless the request can be backed up by some kind of six sigma truth guarantor.

What else can one do?

——————————————————————

This gets to the topic today’s post: Where do new ideas come from?

In my experience they come in intermittent, varied and interrupted intensity, like bees getting called to pollen

Truly blessed is the man or woman who gets to work with people around whom they can think out loud. I have the good fortune of working with people like that, who do not expect your ideas to be fully formed.

Here is an exchange I had with three colleagues (Ted Lewis, Rudy Darken, and Lauren Fernandez) last week about another idea for assessing preparedness. The idea is “half baked,” by which I mean in a non-derisive way, it is not done.

——————————————————————

Me: When you get the chance, can you look at this 6 minute video from David Snowden, the complexity theorist. [The video is posted in this blog, below.] It is a promotional video for SenseMaker software, but it describes a kind of narrative research that could be used by states and cities to assess preparedness.  It basically uses stories as data, and then portrays those data as a (both fixed and dynamic) fitness landscape.

I wonder if we could use an approach like this to measure (or at least portray) preparedness.

**************************

Ted: While you’re at it, take a look at this video (from the United Kingdom, about involving people in their energy future).

It might be a model for doing what you want.

**************************

Rudy — Conceptually, I get what they are doing, but I can’t quite grasp what form this takes or how I would use it [to measure preparedness]. Do you?

**************************

Ted : What is the objective? What do you want to achieve? Then work backwards. If you were going to construct a Maslow Hiearchy for homeland security preparedness, what would it be? That is the goal. Once you have that, what data do you need to collect and display to define or ‘prove’ a layer in this hierarchy? At each layer, drill down to the precepts. What are they.  Keep going until you are exhausted or the listener is no longer interested. Then you are done.

**************************

Rudy — The problem is that the connection between things we do and preparedness outcomes is fuzzy at best. In fact we don’t even know what the metrics are. So that’s what I’d want to use the wisdom of the crowds for. If we could think of a way to get hundreds of responses that help frame the concepts of what goes into preparedness and how much does it matter, you might have a very interesting result.

**************************

Ted — agree! What are the metrics? It could look something like the UK energy site, but what are the levers?

**************************

Rudy — Exactly my point. What if you had a way for the user to say what he thinks the levers are and how he thinks the levers influence outcomes (which he would also have to specify, or pick from a list).

**************************

Me — Unless I’m missing something, the UK app illustrates trade off analysis.  The goal is clear and measurable (achieve 20% of 1990 CO emissions). I think preparedness assessment could lead to trade off analysis, but I think there are some steps missing before we get there — like what the goal is.

As I understand a fitness landscape, it is an image that portrays the relationship between agents and their environment.  Some of those relationships are stable and are unlikely to change without a massive disruption.  So one might say those agents are a good fit with their environment.  Other relationships are unstable and are not likely to persist in the environment (such as paying attention to terrorism in a community that has never had a terrorism incident).  So preparedness would mean something different in each community (or more accurately, each region).  By first mapping the preparedness landscape (by collecting and plotting stories told by agents as described in the SenseMaker software), one would have a baseline from which to look at how preparedness is changing.  This would be analogous to observing how an ecosystem changes as its environment or agents within the system change.

Public choice theory posits that, over time, communities tend to be satisfied with the level of government services they receive.  If they are not, they will tend to elect people who will work to change the level of services — via ending, starting programs; increasing/decreasing taxes.  There’s more to it than that, but public choice assumes a tendency toward a dynamic equilibrium.  I wonder if one can make the same assumption about preparedness: that communities are generally satisfied with their level of homeland security preparedness, otherwise they would take action to change that level.  (This excludes for now one level of a system expecting more or less out of a different level of a system: fusion centers might be an example of that.)

The key to all this, as I understand it, is in one of Ted’s earlier emails: define the data.  I think the objective is agent satisfaction.  The data consist of stories the agents tell about prevention, mitigation, response, recovery, resilience, and the other portmanteau concepts in preparedness – and I think identifying the questions that elicit those stories would be a useful next step.

I want to bring Lauren Fernandez into this conversation.  While she was with DHS, she led pilot projects to figure out how to assess preparedness (in response to HSPD 8 requirements).  She has more experience and knowledge about measuring preparedness than anyone I know.  Here’s what she wrote in an email to me yesterday on this topic:

**************************

Lauren: I watched Snowden’s video, and some of his statements resonated with me—in the real world people share anecdotes, and this information is valuable. I liked the idea of the storyteller also being the interpreter. I didn’t quite understand what a fitness landscape is and how it can be developed.

I do think that different ways of assessing preparedness need to be explored. It is the type of problem where there are not (nor do I think there can be) commonly accepted and measurable indicators across the board. There are some instances where quantitative measures make sense (e.g. average response time of certain equipment in certain geographic area), but this can’t be done for everything. There are instances when people’s perception of performance is adequate, and instances when it is not.

There are so many moving pieces:

- Preparedness requires dollars, and there are differing views on where dollars should be spent and how much the government should be able to acquire and distribute for all needs (education, healthcare, security, etc.) [Bellavita note: this would be a good use of an application similar to the UK energy app.]

- The actors that wish to cause harm in the domestic US are not static, they adapt. Our defenses must also adapt, making measurement difficult.

- Our capabilities to prevent and respond change over time (new technologies, events affect our resources, etc.)

- Some things are just difficult to measure. If you provide enough specification in a question to get a decent answer, that answer is not generalizable. If the question is too generalized, it can’t be answered well.

- We measure because we want to adjust where we place resources (time, money, etc.) People who are a part of the system being measured are motivated to affect the evaluation (it affects programs they care about, personal finances, reputations, etc.). Measuring teacher performance is a good analogy—we end up placing a lot of resources into standardized testing. Teachers spend significant time working to get students to pass the tests. To what end? Has this produced better students that will improve their personal and community life?

When thinking about a new approach to preparedness measurement, perhaps we can start with the end in mind [Bellavita note: to me, the desired end in mind is regional satisfaction with that region's preparedness for certain types of threats] and work backwards to tools that will help us meet that end.

I liked this blog post on the importance of being able to take action. I’m curious if there is any evidence that fitness landscapes can (or could) influence decisionmakers and if the information developed is good enough to make decisions.

**************************

Ted — A couple of observations: You may be describing qualitative reasoning, which is a subtopic of artificial intelligence. No numbers, just directions like up, down, small, big, etc.  Alternatively, your model might simply be “the wisdom of crowds”, and nothing more than a heat map of the US. Ask people to rank their level of satisfaction along a scale.  At least Chris identified the goal: to reach preparedness satisfaction. That is a start. What stories deal with satisfaction?

**************************

Me —  If asked, I could come up with stories about how satisfied I am with my community’s ability to anticipate, respond to and recover from floods.  I could do the same with fires and wind storms.  Those are the major threats where I live, and there are stories people in the community tell each other about, for instance, the wind storm of 1995 that destroyed a lot of trees, knocked out power for a week or so in some places, disrupted travel and so on.

Without actually collecting those stories (yet), I am guessing I could locate on several triangles (like the Snowden video) how prepared I thought my community was to anticipate, prevent, mitigate, communicate about (and so on) a major windstorm. That could provide some measure of satisfaction – or at least a basis for asking. I could also include indicators of likelihood, vulnerability, etc.  Same things with stories about fires, hazardous materials (Interstate 5 runs right through the town), and terrorism  – we have a machine gun store in town (under the right conditions you can rent a machine gun). It’s a few blocks from an elementary school.  There’s a story.

Using an app in Walmart, on iPhones, in schools, in the library, all over the place, people could tell and interpret their preparedness stories; based on the “data points” (still not sure what that means) we would get a fitness landscape.  What that landscape means - and what its action implications are – is also unclear to me.

But the task is to collect and display stories.  Then let’s see what emerges.

**************************

Ted — And would you open this up to the crowd? You still need some way to rate or compare “stories”.

**************************

Me – I would open it up to the crowd — i.e., US residents — the people who are supposed to benefit from all of this preparedness.  As I understand the SenseMaker model Snowden uses, the designers create the categories to be rated. The story tellers are the ones who rate their own stories.  The video describes the process, from the 1:13 mark to the 4:15 mark.

And to emphasize again, I’m stumbling around here.  If preparedness is a complex idea and activity, maybe a tool sensitive to complexity could help give a picture of preparedness (I hesitate to say “measure preparedness”) in a way that would allow policymakers/analysts to try to explain what accounts for the different preparedness landscapes and whether a particular landscape is politically, fiscally, socially acceptable….  (and here I trail off into mumbling).

**************************

Lauren — There are a number of studies that focus on trying to evaluate the preparedness of citizens.  (DHS tracks this research here: http://www.citizencorps.gov/ready/research.shtm).  Although there are ‘rate your community’ surveys for general attributes, I am not aware of studies that focus on citizens evaluating the preparedness of their communities.

Some questions to consider (meant not to attack this idea, but rather to strengthen it through critical thinking):

- Do citizens know enough to develop a good picture?  (Perhaps ‘enough’ is the wrong question, as different individuals will have different pictures which in aggregate is valuable, but do they have key insight.)

- Do citizens know how their emergency services prepare?  In my mind, knowing that the fire and police chiefs talk over pancakes once a month (as they apparently do in Austin, Texas) would make me feel more prepared than having a spiffy new EOC.  I know we have a spiffy new EOC in [my city], but I don’t know what the relationship between fire and police is like.

- What would and could policymakers do with this information [about preparedness]?  (Would they shut down the machine gun rentals near the elementary school?)

- I think Rudy’s statement that the “connection between things we do and preparedness outcomes is fuzzy at best”  is very true.   We have pulled multiple levers in the past 10+ years (fusion centers, WMD response equipment, Citizen Corps, etc.) but don’t have a good handle on the delta of preparedness.   Is there a good way to determine the relationship to outcome, or are the events we are preparing for so rare and complex that we can’t do this well?

- That said, there may be a few exceptions in understanding the connection of actions to preparedness. We’ve learned that people don’t want to evacuate without their pets.  Seems trivial, but by satisfying this behavioral factor by planning to shelter pets, I think (and perhaps there is evidence) that preparedness is improved.  What other behavioral keys exist?  What makes people feel comfortable reporting suspicious behavior?  Could a good outcome of a study be to encourage more study on the impact of levers first before developing standards?

-  What would and could citizens do with this information?

-  Are citizens rational actors?   Recent developments in behavior economics have found that our assumption that investors are ‘rational actors’ is not a good one.  I wonder if preparedness satisfaction might be very tied to psychological factors.  Or would public choice theory ring true?

- Should communities be preparing for something greater than what they have experienced?  Stories from the Midwest may relay a high level of satisfaction, but the New Madrid fault hasn’t been shaking.

Mumble, mumble…

**************************
Ted —  Most of your questions boil down to the question, “do you trust the wisdom of crowds”?

Is perception about preparedness reality? Probably.

Fukushima illustrates this. Nuclear power is just as safe ( or not) now as it was before the earthquake, but I bet the landscape of preparedness satisfaction is radically different now than before!

**************************

Rudy – This is fairly well outside my area of expertise, but from what I’ve seen, true preparedness is unachievable without large-scale community (i.e. private citizen) involvement. That implies some level of ownership of the problem. There currently is little to none of that. It would be interesting to ask a large swath of the population how likely they think their neighbors would be to come to their aid if they needed it — like when a tornado rips through a neighborhood, damaging some homes but not others. Then how likely would they be to aid their neighbors. Then broaden to include the role of responders. I wonder if the fitness landscape for preparedness is a good idea but one that needs to be built in incremental layers, not all at once.

**************************
Ted — Is it possible to build a landscape out of the words people use to tell their stories? What if nobody comes to our field of dreams?