Shane Harris at the National Journal has the latest in a long string of excellent stories on intelligence issues in this week’s edition, only available right now by subscription, but likely to turn up on the National Journal’s sister publication GovExec in the next few days. (Update: now available here). The story looks at the parallel and converging efforts of John Poindexter’s Total Information Awareness program and Michael Hayden’s related efforts at the NSA in exhaustive detail, telling a complex story of patriotic intent, bureaucratic rivalry, technology challenges, and political pressures.
The story paints a fairly sympathetic picture of Poindexter’s TIA program, highlighting its speed and creativity:
In February 2002, Poindexter established a secure, classified computer network for testing analysis software and tools that might be worked into TIA. As the system came together, this experimental network would be the engineers’ Bonneville Salt Flats, a place to test-drive the state of the art. If tools passed muster there, they might end up in the design Poindexter had in mind.
“If there was a vendor with some great gizmo, they’d have to go through an arduous one- or two-year process to get that accredited by an intelligence agency,” said Robert Popp, who was the No. 2 TIA official and Poindexter’s deputy. “That didn’t fit our parameters. We wanted to kick around these various technologies to see their utility. The network could put it through that whole two-year process in a few months.”
And it describes an early real-life application of TIA:
As months passed, more agencies joined, and some began using TIA for real intelligence operations. For instance, in 2003 the Pentagon’s Criminal Investigation Task Force, which was established to fuse law enforcement and intelligence techniques in fighting terrorism, was interrogating detainees at the U.S. military facility at Guantanamo Bay, Cuba. Stacks of interrogation reports piled up, and the interrogators struggled to make sense of the information they contained. Some detainees frequently mentioned the same names or places. Some detainees claimed to know each other. Others didn’t. The interrogators turned to the TIA network to help sort out the hundreds of reports and potential leads.
“They provided the interrogation reports to analysts, and [the analysts], using several link-analysis tools provided by TIA, tried to discover interesting nonobvious relationships,” Popp said. Link analysis detects connections between people through common associates or backgrounds, and creates web-like diagrams of the connections. “The link-analysis tools showed the interrogators things that were not apparent to them — very valuable, useful information that they could then use in follow-up interrogations.” Popp said that the investigators also knew after they concluded their interrogations that some detainees were not terrorists, so those reports were used to create a sort of baseline for what a nonterrorist looked like. The tools could then be calibrated to disregard certain attributes and search for others that were salient, Popp said.
This is the appropriate and effective way to use social network analysis: as a tool for deciphering links among hundreds (or thousands) of people, not hundreds of millions.
But in mid-2003, TIA was derailed due to public scrutiny and a Congressional block on funding, and elements of it migrated over to the classified domain. (TIA had been unclassified by design, as a way to involve a wide spectrum of smart people and drive innovation). But some parts of TIA didn’t migrate:
But it discontinued some programs, most notably a multimillion-dollar effort to build privacy-protection technologies. ARDA also abandoned the effort to build audit trails in TIA, which would have permanently recorded any abuse by users.
As I’ve said before, it was a big mistake to shut down TIA; its shift to the classified domain has eliminated privacy protections, made oversight more difficult, and probably slowed down the pace of innovation.
Harris then uncovers the new name for the TIA network:
The experimental network’s name was changed from TIA, to erase any connection to its past. Today it’s called the Research Development and Experimental Collaboration (RDEC, pronounced ARdeck). The NSA is the biggest player, with at least 15 nodes as of December 2004, according to official documents. “I think it’s considerably more today,” said a former government official knowledgeable about RDEC. A spokesman for the NSA said he had no information to provide about the network.
….The Defense Intelligence Agency, which like the NSA is overseen by the Pentagon, is one of the largest RDEC users. In an interview, Lewis Shepherd, the chief of the agency’s Requirements and Research Group, said that RDEC is “the most successful attempt at bringing together a wide variety of analysts and agencies to work and think outside of the box collaboratively,” specifically on counter-terrorism. “[It] opens access to a variety of data sources to different tools that haven’t been able to access that data.” For example, RDEC lets analysts conduct repeated keyword searches on many different data streams, Shepherd said. It “sparks out-of-the-box innovation in how we do information-sharing.”
It’s difficult to assess RDEC beyond what’s discussed in the story; if it is an effective intelligence tool, and if it’s consistent with U.S. law, then it has a legitimate purpose and necessary role in the war on terror. But the apparent lack of accompanying privacy protections and audit trails concerns me; the NSA should take steps to add these features to the network, assuming that they don’t currently exist.
There are many other interesting details in the paper – as mentioned earlier, I’ll post an updated link to the full text if and when it appears online.
(Hat tip: Noah).