Homeland Security Watch

News and analysis of critical issues in homeland security

October 29, 2009

Homeland Security’s War On Subjectivity

At first I thought they were being funny.

“Let’s hold a hearing about measuring preparedness and do something wacky to see if anyone’s paying attention.  Let’s attack something for being subjective.  And use subjective data to support our claims.”

On Tuesday, the Subcommittee on Emergency Communications, Preparedness, and Response held a hearing titled “Preparedness: What has $29 billion in homeland security grants bought and how do we know?” [You can read the prepared testimony and watch a video of the hearings here].

During the hearings, members of Congress criticized DHS for using a particular assessment tool — something called “Cost to Capability.” Otherwise known as C2C

The primary criticism: the tool was too subjective.

And what data were offered to demonstrate how subjective the tool was?

Opinions of the committee members.  Opinions of witnesses. Opinions of constituents.

Two examples (with my emphasis):

The tool “… remains entirely subjective. Grantees are simply asked to guess the impact of the grants on their grants.” said Chairman Henry Cuellar (according to his prepared remarks).

“I am … concerned that the tool requires a subjective judgment of our base capabilities and perhaps more importantly how much an investment has increased a capability,” said the unfailingly gracious Dave Maxwell, Director and State Homeland Security Adviser of Arkansas’ Department of Emergency Management.

Were Cuellar and Maxwell offering objective or subjective assessments?  Aren’t opinions — even informed opinions — subjective?

If they were being subjective, what’s so bad about subjectivity?  Why do intelligent, well-meaning people seem to be at war with it?
————————————————–
Tuesday’s hearings were not a joke.  Anyone interested in this topic can see a dozen thoughtful, knowledgeable and concerned people struggling over this very wicked problem: “What has $29 billion in homeland security grants bought and how do we know?”

Those are important questions.  They seem ripe for what FEMA Deputy Administer Tim Manning later described as “rigorous analysis and the development of precise metrics which will enable us to connect dollars spent to results achieved and ultimately to improvements in preparedness.”

How hard could it be to figure out some objective way to answer those questions?

“I thought C-2-C was supposed to get rid of the guesswork.” said Chairman Cuellar.
————————————————–

No wonder he thought that.  Here’s how C2C was marketed:

With the tools generated by the C2C Initiative, [homeland security] grantees will be able to maximize their local preparedness investment strategy with respect to the Nation’s Homeland Security priorities. By design, these tools will adapt to changes in the Nation’s Homeland Security Strategy, translating national priorities into a clear prioritization of capabilities-based investments that all levels of government can use. C2C tools will inform grantees’ use of limited grant funding and better measure how grants increase the capability of States and local communities to respond to all-hazards.

Tools, maximize, preparedness, strategy, adapt, priorities, prioritization, capabilities-based, investment, limited funding, better measures, all-hazards — one can hear buzzers going off all over the place.

Whatever C2C is supposed to do, one is comforted by the promise that it will provide tools.  Tools are things we can hold in our hands.  We use tools to fix things.  So C2C will fix things.  It will fix the thorny problem of how to measure preparedness.

————————————————–

I do not want to add to what Cuellar described at the end of the hearings as “a tsunami of concerns” about C2C.  In my view, C2C is only the latest iteration of the quest to measure preparedness — the homeland security equivalent of trying to turn straw into gold.

In 2003, Homeland Security Presidential Directive 8 commanded that there be an annual status report of the nation’s preparedness.

I do not believe that command was ever obeyed.  But I do know of a half dozen pilot efforts to figure out how to do it.

In 2006, the Post-Katrina Emergency Management Reform Act reissued the command as a requirement to establish a “comprehensive system to assess, on an ongoing basis, the Nation’s prevention capabilities and overall preparedness.”

Three years later, from C 2 shining C, the nation still waits to find out. (I am sure I am not the first to make that horrible pun. Still, I do apologize.)
————————————————–

Congressman Cuellear wrote his 1998 doctoral dissertation on government performance: “A Comparative Analysis of Legislative Budget Oversight: Performance Based Budgeting In The American States.” He volunteered to help FEMA and offered what he called “free advice” (on a single piece of paper) about how to measure preparedness: figure out the mission and what preparedness means, determine your goals, your strategy, your performance measures….

Really smart people have worked on this wicked problem for six years. If it were as straight forward as Dr. Cuellear suggests, perhaps it would have been done already.

That national preparedness hasn’t been measured – in spite of major efforts to do so — ought to count as objective data to support the hypothesis that perhaps it cannot be done.
————————————————–

Cueller said (regarding C2C) “It’s better to not defend something that’s not working.”

Maybe those words should be directed to those who believe we can measure preparedness objectively.

————————————————–

Can we try something different?

Perhaps an approach could be built around blatantly subjective impressions of preparedness.  The impressions would come from the professionals charged by their constituents with the mission to make sure their communities and the nation are prepared and resilient. Maybe we can trust that — in general — they know what they are talking about.

————————————————–
Here’s an example: One of the witnesses was Kathy B. Crandall, Director of Homeland Security & Justice Programs from the Columbus, Ohio Urban Area.  She said, in response to a question, that “Yes, we have solved the [interoperable communications] problems in the Columbus urban area.”

You read that correctly.  She actually said the interoperability problem in her jurisdiction had been solved.

Was she being subjective?  I think she was. She offered no objective data to support that claim.

But I believed her.

Should other people believe her?  Can we build a national assessment program around trusting the “word” of people in a position to know whether or not they are prepared?

But what if the people we ask have other motives?  What if they are just in it for the money and will say whatever they think they need to say?  What if they lie?

On the other hand, what if they are giving as accurate a picture of preparedness as we are ever likely to get?

————————————————–

The War on Subjectivity is fundamentally about not trusting each other.  I can’t trust what you tell me because, being human, you probably have some hidden motive for what you’re saying.  Or worse, you may not even know what you’re talking about.

And you can think the same about me.

————————————————–

Preparedness — as the participants in Tuesday’s hearings acknowledged in one way or another — is not an “external object” subject to the same kinds of measurement processes as physical objects.  Preparedness – like the resilience Phil has been writing about this week – is a subjective concept.

We can objectify it as much as we care to.  But like the term “homeland security,” preparedness means different things to different people.  That’s not likely to change anytime soon.
————————————————–

So what has $29 billion in homeland security grants bought and how do we know?

Tim Manning provided a compelling answer:

Intuitively, we could answer the question “Are we better prepared?” with a “yes.” We could validly point to the amount and type of equipment that has been purchased, the physical security improvements that have been made, and the planning and training improvements that have occurred, and conclude that we are better prepared. Our national, state, local, tribal and territorial efforts have certainly increased our interagency planning across the spectrum of preparedness. This is in itself an achievement that greatly improves our ability to act decisively in a crisis.

I think it’s more than intuition that allows one to say that.  Talk with any experienced practitioner in a homeland security-related activity and ask if we are better prepared for a variety of events than we were on September 10, 2001.

When I ask that question, the subjective but consistent response is almost always yes.

Ask the same question about equipment, training, collaboration, information and intelligence sharing, border and transportation security, immigration, interoperability, critical infrastructure protection, private sector preparedness, attention to privacy and civil rights concerns, terrorism finance, weapons of mass destruction — pick your list.

Are we better prepared now than we were on September 10, 2001?

When I ask, the answer is almost always yes.

————————————————–

Don’t get me wrong.  There is still a lot to do in all of these areas.  I have objective evidence to prove that assertion.

And while we’re at it, the preparedness grants I want for my community ought to get priority attention from Congress, DHS and FEMA.

But that’s an argument for a different hearing.
————————————————–

Our country has enough wars going on right now.  We do not need another one.

I propose we open negotiations with Subjectivity.  We should seek at least a tentative peace with it.

I believe once we trust — and verify — what Subjectivity has to say, we will learn that we are an increasingly prepared nation. And homeland security can move forward from there.

Of course that’s just an opinion.

Share and Enjoy:
  • Digg
  • Reddit
  • Facebook
  • Yahoo! Buzz
  • Google Bookmarks
  • email
  • Print
  • LinkedIn

3 Comments »

Comment by William R. Cumming

October 29, 2009 @ 11:00 am

WOW! Terrific post. By the way OMB has a much better (IMO) reporting report format than the long outdated and inadequate document that the Preparedness Directorate inherited from the Department of Justice which has a mostly fire and forget attitude towards its grants program as just a political effort not really designed for effectiveness. Planning and Evaluation at DOJ has been dead for the last 5 adminstrations and also ethics charges filed seveal times against DOJ officials who lined up grants for their personal employment before departing DOJ. Essentially as readers know I would get DOJ out of administering programs, functions, and activities that do NOT involve the criminal law and its prosecution and prison system. DOJ is over 100,000 people and almost half have some kind of law degrees. In fact it is where many law school graduates go who do not actually wish to practice law. Why the diatribe. Because many of the DOJ grant people working homeland security ended up being transferred to DHS and even FEMA. Many second careerists who love paycheck more than accomplishments.
Okay so what is objective and what is subjective. Note the final Target Capabilities List was issued in September 2007! What might be done is find out how many of those target capabilities have been accomplished? And capabilities is an important word, very important! Much more important than subjective or objective because whatever the vision of the observer or evaluator capability means that it can and will be done, not just reasonable assurance like under FEMA/NRc REP program–see 10 CFR Part 50 and 44 CFR part 350! Appendix E of 10 CFR Part 50 has long contained the word “verify” and this word however defined means the capability exists. Will how about some verification (although the REP program requires it never done in fact)for these HS Grant programs? Do you know how very very tough it is to make sure an actual capability exists 24/7 and 365 days a year? Just ask the Armed Forces! So let’s get serious! CNN this week revealed American Samoa received and spent $13 Million for preparedness and accomplished nothing, nada, and why no waste, fraud, and abuse from that debacle?
Chris’ post raises several other thoughts but be back later. I always did like President Reagan’s “Trust but verify” notion. Peggy Noonan?

Comment by William R. Cumming

October 29, 2009 @ 2:10 pm

Chris! Maybe just missed it but could you indicate the origin of C2C and what guidance document spells it out or requires its application to FEMA/DHS grants? When was this “strategy” adopted?

This term is brand new to me but probably just missed its announcement in a DHS or FEMA press release!

Hoping you can help? And by the way do the grant recipients know about C2C?

Pingback by Homeland Security Watch » “There is a quality even meaner than outright ugliness”

April 12, 2011 @ 5:22 am

[...] I wrote in this blog two Octobers ago, there have been at least 6 well-funded pilot efforts to figure out how to measure preparedness. [...]

RSS feed for comments on this post. TrackBack URI

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>