CIO Magazine has an excellent article in its latest issue on the role of data mining in homeland security and counterterrorism, assess its benefits and limitations, considering the types of tasks for which it is useful, and calling for greater rigor in determining when and where it should be utilized. After noting the proliferation of data mining projects in the federal government since 9/11, the article notes:
But some experts are beginning to question whether an IT strategy of unlimited scope, budget and schedule will best serve that end [of protecting the country].
….”No one [in the government] has looked at data mining from an IT value perspective,” says Steve Cooper, former CIO of the Department of Homeland Security. “I couldn’t figure out [the value of data mining] when I was in DHS, and I can’t figure it out now. But that didn’t stop us from using it.”
In other words, according to Cooper, no one has done a business case analysis to determine whether the government is getting a return on its investment. Instead, a rationalization is usually sufficient: If a project has a chance to catch just one terrorist, then it is worth it.
This is a somewhat surprising admission from the former CIO of the Department of Homeland Security, who was in the trenches when a number of relevant programs were being considered.
The story then puts data mining projects into two categories: subject-based systems (i.e. link analysis) and pattern-based systems. It offers examples of useful data mining projects in each category (links among Gitmo prisoners, patterns of activity among cleared DOD personnel to detect counterintelligence), but only in cases where the population that is being analyzed is at a relatively modest scale, i.e. thousands instead of millions. It then takes a long look at TSA’s CAPPS-II and Secure Flight programs as examples of programs that have thus far failed to achieve their objectives because of immodest scope and an unclear business case:
Capps II and Secure Flight had no such ROI mechanisms. But rather than reexamine the goals and scope of the projects, the government simply expanded them to include profiling, a hunt for common criminals and more. And as happens so often with IT projects when their goals are too broadly defined, the system is still not active despite an originally planned go-live date of November 2003.
“TSA was never willing to reevaluate the scope of the project,” says Jim Dempsey, policy director of the Center for Democracy and Technology, who was part of the TSA’s Secure Flight Working Group with Schneier. “So now, five years after 9/11, we still don’t have an automated system for matching passenger names with names on the terror watch list. Civil liberties had nothing to do with that.”
The story concludes with a general plea for greater oversight of government data mining activity, as a way to ensure that programs actually deliver real security dividends:
Most data mining projects are not subjected to a rigorous business case analysis. Two current intelligence CIOs who were otherwise unable to comment for this story agreed that this is an issue that they struggle with. The DoD’s Technology and Privacy Advisory Committee (TAPAC) developed a 10-point system of checks and balances [Ed. note: see page #’s 54-55 of this document] that it recommended every agency head apply to data mining projects, but Cate says that it has never been implemented. Similarly, the National Academy of Sciences recently appointed a committee to develop a methodology that the government can use to evaluate the efficacy of its antiterror data mining projects, but the target date for its report is still more than a year away.
What’s left is the status quo. That’s troubling to people like Cate. “There are some extraordinarily smart people [working on data mining systems], and I would be hard pressed to think that they are wasting their lives on something that doesn’t work,” he says. “But one of the things [TAPAC] kept focusing on was that you have to be able to show that it works within acceptable parameters,” a responsibility that he says rests with agency heads.
Agency heads aren’t accepting that responsibility, says Cate. “As far as the oversight process is concerned, it is clear that [data mining to prevent terrorism] is a disaster.”
These efforts to develop consistent standards, methodologies, and checkpoints for data mining projects are very important, as I’ve suggested in previous posts. Data mining can deliver real value to the war on terror, but it needs to be developed in a much less scattershot manner than has typically been the case over the last five years.
Overall, a very good story, one that usefully clarifies the role and typology of data mining for homeland security and counterterrorism.