Boy meets girl. Boy loses girl. Boy gets girl back.
That doctrine guides innumerable novels and movies. In the right hands, the formula can be used to create a film as sublime as Casablanca. In the wrong hands you get something like “Harold & Kumar Escape from Guantanamo Bay.”
Homeland Security owns an equally well known formula: “Risk is a function of threat, vulnerability and consequence.” Used in the right hands (see Bob Ross’ comments in the previous post), the formula can be a helpful way to structure thought.
The formula can also produce risk assessments.
Risk assessments are stories — in the good sense of that word — used to justify spending money for some activities rather than others. The generic story is built around threats to harm something, ways the something can be harmed, and the results if that something is harmed. The moral of the story is “If you give us money, we will mitigate the risk.”
Good homeland security risk assessment stories are not easy to write.
Risk based anything is a cornerstone of the illusive homeland security doctrine. Here’s the assertion in the 2007 National Homeland Security Strategy: “We must apply a risk-based framework across all homeland security efforts in order to identify and assess potential hazards …, determine what levels of relative risk are acceptable, and prioritize and allocate resources among all homeland security partners….“ Here’s the 2009 formulation from the Obama administration’s homeland security agenda: “Allocate Funds Based on Risk: Allocate our precious homeland security dollars according to risk, not as pork-barrel spending or a form of general revenue sharing. Eliminate waste, fraud and abuse that cost the nation billions of Department of Homeland Security dollars.”
The risk acolytes sometimes sound like one of the characters in Harrold and Kumar: “If you want to know the secret of being, you’ll come with us.”
Seven and a half years after the onset of the Overseas Contingency Operation (nee GWOT), there remains a substantial gap between the theory and the practice of the risk formula. A June 2008 GAO report concludes DHS “reasonably” uses “empirical risk analysis and policy judgment” to allocate grants. (See another helpful GAO report on this issue here.)
I read “empirical risk analysis” to mean DHS used numbers. Where do those numbers come from?
I read “policy judgment” to mean politics and opinions were involved. Who is making those “policy judgments,” and how do people outside Washington get in on that judgment?
I read “reasonable” in the GAO report to be an acknowledgement that determining risk is really hard; DHS has done they best they can with the data available and the constraints they face; they are better at doing this than they used to be; and they are continuing to improve.
But is it acceptable inside the “fierce urgency of now” to wait 7 years to get an equation right?
What to do about this? How is it that homeland security came to place so much faith in the logic of risk and its conceptual offspring? Are there alternatives we can try?
Perhaps we could acknowledge homeland security risk assessment’s dependence on a falsely precise empiricism, and experiment with allocating resources explicitly (and with accountability) on the basis of opinion and politics. Or maybe we could try an allocation strategy built around the “redundancy, flexibility, and diffuse control” options Geerat J. Vermeij discusses in Natural Security: A Darwinian Approach to a Dangerous World? (Edited by Sagarin and Taylor, and available here.)
Another approach is to blow off such alternatives (what’s a nice word for naive?), and work harder at “solving” the problem.
I came across a numbingly-titled but intellectually honest paper called: “Incorporating Assessments of Terrorism Risk in Homeland Security Resource Allocation Decision Making: Closing the Gap Between Current and Needed Capabilities.” Its 10 pages are worth reading by anyone who cares about homeland security and risk, and who believes “the problem” is fixable. (The paper is available here; it appears to have been written to support an April 13-16 conference called “Risk-Informed Decision Making for HLS Resource Allocation;” information about what looks to be a thoroughly interesting conference can be found here).
Some excerpts from the paper [and my overly simplified interpretations]:
“Despite the best efforts of numerous experts from the government, industry, and academia, fully effective and transparent integration of risk assessments into DHS homeland resource allocation decision making remains an elusive goal.” [The theory is not working in practice.]
“The risk construct … [Risk = Threat x Vulnerability x Consequences] is logical, intuitively appealing, and consistent with conceptualizations of risk used in other domains. However, uncertainty inherent in deriving estimates for its components in the case of terrorism risk continues to compromise its usefulness in DHS resource allocation decision making. As a result, terrorism risk assessments have not played the prominent role they were expected to play in DHS resource allocation decision making. More robust and defensible methods for generating required inputs for this terrorism risk construct are required if it is to become an important factor in homeland security resource allocation decision making.” [The theory sounds good, but the data to use it aren't there.]
“Significant perceived shortfalls in our current ability to generate defensible estimates of the threat of terrorism, terrorism vulnerability, and the consequences of terrorism continue to hamper DHS’ ability to effectively incorporate terrorism risk assessments into its resource allocation decision making process. In addition, predicting and/or measuring the risk reduction effect or payoff of terrorism risk management initiatives remain a complex and unresolved challenge.” [We can't defensibly calculate the values the formula needs; we also can't figure out the effect of reducing risk.]
“[The] statistically oriented, historical frequency approach [to determine threat] is useful for that portion [of the] DHS’ “all hazards” mission space associated with natural catastrophic and accidental events. However, [that analytical approach] does not apply to the threat (i.e., probability of attack) of terrorism. Risk analysts assessing the risk of terrorist activities are not faced with accidental random loss-inducing outcomes or acts of nature. They must deal with the potential for harm and/or loss deliberately inflicted by intelligent and adaptive adversaries willing to operate outside normally accepted patterns of behavior. Simply identifying all potential terrorists is a monumental challenge. Predicting their behavior – i.e., which targets or kinds of targets terrorists may strike, how they will choose to strike (i.e. weapons and/or threat vector), and when they will strike is an even more daunting and uncertain element of terrorism risk assessment.” [Terrorism is a really different kind of threat.]
“Assessing the vulnerability of potential terrorism targets might seem to be a theoretically more tractable problem. However, it requires detailed knowledge of the target, the nature of the terrorist attack, and the circumstances under which it will occur. Thus, vulnerability assessment involves many of the same uncertainties encountered when assessing the intentions and capabilities of potential terrorist adversaries. Even where credible estimates are possible, the sheer magnitude of the task of assessing the vulnerabilities of every potentially lucrative terrorist targets in our open society is overwhelming.” [Everything is vulnerable to something.]
“Predicting the consequences of terrorism events also presents difficult intellectual, philosophical, and emotional challenges. Most previous catastrophic risk assessments have limited their prediction and/or measurement of consequences to direct effects which can usually be estimated in relatively straightforward, transparent, and reliable ways. These direct effects typically include human fatalities, injuries, and the direct economic cost of the physical damage inflicted. However, it is widely acknowledged that terrorism events usually result in a broad range of secondary or indirect effects or costs that can be more extensive, and often more important and long lasting….Not only are these secondary or indirect effects of terrorist events difficult to predict or measure, but knowing how to prioritize them, and/or to compare them with the consequences of catastrophic accidents and/or natural disasters remains an unanswered question with complex societal and emotional dimensions.” [Consequence streams approach singularity.]
Risk Mitigation Effectiveness/Cost-Effectiveness:
“Assessing the effectiveness or payoff of a terrorism risk mitigation measure might initially seem to be a reasonably straightforward challenge. However, the effectiveness or cost-effectiveness of any terrorism risk mitigation initiative is inevitably dependent upon, and confounded by the way terrorists react to that measure…. Related closely to the issue of predicting the effectiveness or cost-effectiveness of risk mitigation measures is deciding how much our Nation should be willing to allocate to terrorism risk reduction, or how great a risk of terrorism we should be willing to accept. In reaching this judgment it must be recognized explicitly that resources allocated to reducing the risk of terrorism impose an economic cost on the Nation since they are unavailable for other needs. Thus, anti-terror initiatives represent other opportunities foregone and must be evaluated as such.” [Perhaps money allocated to homeland security could help the Nation more if it were used for something other than homeland security.]
Maybe risk can be salvaged as a way to allocate homeland security resources effectively. If you are in that camp, I think you will find the paper — and the questions it contains for the conference panels — intriguing and thoughtful.
The problems we have experienced trying to get homeland security risk assessments correct might also signal an opportunity to try something fundamentally different. Maybe we have traveled long enough down the risk assessment path.
Captain Renault: What in heaven’s name brought you to Casablanca?
Rick: My health. I came to Casablanca for the waters.
Captain Renault: The waters? What waters? We’re in the desert.
Rick: I was misinformed.