According to the Government Accountability Office (complete report, 101 pages, available):
The Department of Homeland Security (DHS) lacks reliable information about BioWatch Gen-2’s technical capabilities to detect a biological attack and therefore lacks the basis for informed cost-benefit decisions about upgrades to the system. DHS commissioned several tests of the technical performance characteristics of the current BioWatch Gen-2 system, but has not developed performance requirements that would enable it to interpret the test results and draw conclusions about the system’s ability to detect attacks. Although DHS officials said that the system can detect catastrophic attacks, which they define as attacks large enough to cause 10,000 casualties, they have not specified the performance requirements necessary to reliably meet this operational objective. In the absence of performance requirements, DHS officials said computer modeling and simulation studies support their assertion. However, none of these studies were designed to incorporate test results from the Gen-2 system and comprehensively assess the system against the stated operational objective. Additionally, DHS has not prepared an analysis that combines the modeling and simulation studies with the specific Gen-2 test results to assess the system’s capabilities to detect attacks. Finally, we found limitations and uncertainties in the four key tests of the Gen-2 system’s performance. Because it is not possible to test the BioWatch system directly by releasing live biothreat agents into the air in operational environments, DHS relied on chamber testing and the use of simulated biothreat agents, which limit the applicability of the results. These limitations underscore the need for a full accounting of statistical and other uncertainties, without which decision makers lack a full understanding of the Gen-2 system’s capability to detect attacks of defined types and sizes and cannot make informed decisions about the value of proposed upgrades.