Homeland Security Watch

News and analysis of critical issues in homeland security

October 23, 2012

Homeland security’s exabyte problem

Filed under: General Homeland Security — by Christopher Bellavita on October 23, 2012

“There were 5 exabytes of information created by the entire world between the dawn of civilization and 2003. Now that same amount is created every two days.”

That’s a quote attributed to Google’s chief executive officer.

An exabyte is one quintillion bytes.  I can’t get my brain around one quintillion, let alone five of them.  I fear zettabyte (one sextillion bytes) is right around the corner.  I don’t know what any of that means. But I’m persuaded it’s a very big number.

Secretary Napolitano has talked about “the big data problem” —  the volume, variety, velocity and veracity of data generated by homeland security activities exceeds the ability of the enterprise to understand what the data means.  Hence “big data.”


Writing in November’s issue of Foreign Policy, Uri Friedman offers a brief history of how we got to the big data problem. Here are some excerpts, liberally adapted from Friedman’s article.


  • 1887-1890: Modern data processing age begins. Herman Hollerith invents a machine that reads holes punched into paper cards.
  • 1935-1937: Passage of the Social Security Act initiates government action to start keeping records on 26 million Americans and 3 million employers.
  • 1943: The British code breakers at Bletchley Park invent Colossus, the first programmable electronic computer; Colossus can read 5,000 characters a second.
  • 1961: The U.S. National Security Agency starts using computers to collect and analyze signals intelligence.
  • 1965-1966: The national government considers transferring all government records — including over 700 million tax records and almost 200 million finger prints — to a single data center.
  • 1974: The 1974 Privacy Act is enacted, limiting the personal information government can share.
  • 1989: Tim Berners-Lee develops the idea of the World Wide Web. “The information contained would grow past a critical threshold…so that the usefulness [of] the scheme would in turn encourage its increased use.” Or, “If you build it, they will use it in unimaginable ways.”
  • 1996: Bill Clinton claims “We are developing a supercomputer that will do more calculating a second than a person with a handheld calculator can do in 30,000 years.”
  • 1997: Michael Cox and David Ellsworth use the term “big data” for the first time. “Data sets [they were working with] are generally quite large, taxing the capabilities of main memory, local disk, and even remote disk.… We call this the problem of big data.”
  • 2002: John Poindexter leads a Department of Defense effort to combine government data sets into one “grand database” that would sift through communications, criminal, educational, financial, medical, and travel records to identify suspicious people.
  • 2004: The 9/11 commission calls for a unified network-based information and intelligence sharing system.
  • 2007 – 2008: Social networks proliferate. Wired magazine writes about the end of theory, “a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear” analyzing and understanding the data generated by the explosion of network activities.
  • 2009: The Indian government approves a plan to fingerprint, photograph, and take an iris scan of its 1.2 billion people, and assign each person a 12 digit identification number, creating the world’s largest biometric database. The nominal purpose of the program is to “improve the delivery of government services and reduce corruption.”
  • 2009: The Obama administration starts data.gov in support of its open government initiative. The website reportedly has more than 445,000 data sets.
  • 2009: the United Nations announces plans to create an alert system that captures “real time data on the impact of the economic crisis on the poorest nations.” The aim is to predict “everything from spiraling prices to disease outbreaks by analyzing data from” mobile phones, social networks, and related sources.
  • 2010: Each day, the national security agency intercepts and stores over 1.7 billion emails, phone calls and other communications. Walmart claims to hold over 460 terabytes of information about its customers shopping and related habits.
  • 2011: IBM’s Watson computer defeats two humans on the television show Jeopardy. The computer system can scan 200 million pages of information – that’s four terabytes of data – in a few seconds.
  • 2012: The Obama administration announces a “big data research and development initiative” to respond to a US government report that calls for every federal agency to have a big data strategy. The National Association of State Chief Information Officers makes the same argument for state agencies.
  • 2012: Facebook has more than 900 million users, who post 300 million photographs every day, along with 3.2 billion new comments and “likes.”
  • 2012: Hillary Clinton announces a public-private partnership called “Data 2X” to collect statistics on the political, economic, and social status of women and girls around the world. “Data not only measures progress – it inspires.… Once you start measuring problems, people are more inclined to take action to fix them because nobody wants to end up at the bottom of a list of rankings.”
  • 2018: Everyone knows everything all the time about everyone else, but there remains a great deal of confusion and uncertainty. (I made some of that up.)


Meanwhile, back in the King James version of Ecclesiastes, the poet writes about the big data problem this way,

“And further, my son, be admonished by these: of making many books there is no end; and much study is a weariness of the flesh.”

Or maybe — to push the risk of triggering a Palin response — the big data problem is not as new as the buzz suggests.

“The thing that hath been, it is that which shall be; and that which is done is that which shall be done: and there is no new thing under the sun.”

I am naively optimistic humans will — somehow — learn how to engage with big data, as we did with the piddling 285 terabytes of data produced by the printing press.

Share and Enjoy:
  • Digg
  • Reddit
  • Facebook
  • Yahoo! Buzz
  • Google Bookmarks
  • email
  • Print
  • LinkedIn


Comment by Philip J. Palin

October 23, 2012 @ 4:58 am

Bait taken:

There has always been sufficient cause for distraction. There have always been plenty of shadows dancing along the back of the cave.

Questions matter more than information. The insightful, authentic question will attract information that matters.

Gadamer wrote, “To question means to lay open, to place in the open. As against the fixity of opinions, questioning makes the object and all its possibilities fluid.”

Comment by William R. Cumming

October 23, 2012 @ 5:34 am

So why are the controls of federal government statistics largely housed in OMB when few agencies or departments even have Chief Statisticians?

Of course needed are better filters to screen out background “noise” data!

Are blogs adding noise or analysis or synthesis that keeps focus on critical issues and events?

Why is important to defend our Homeland or provide its security?

Comment by DHS: Exabyte challenge or Breach of Faith - Treason - at the Executive level

October 23, 2012 @ 11:50 pm

We here on “Main Street USA” w/numerous limitations have no problem whatsoever seeing the intentional – breach of faith – towards – We the People – an oath of office you swore to and to the Constitution by you, this supposed commander-in-chief you refer yourself to and so willing to perpetuate a lie to distract from relection efforts as supported by timely report where this WH and the Secretary of State knowing full well the ongoing attack on our Embassy and willingly curtailing security presence to safe guard our US Ambassador…well, why is is that from the first hours of attack report we stated clearly, “Treason I Say and Shame on You I Say” – yes, treason and nothing less calling for an immediate and full Congressional Hearing for a – breach of faith – where We were very able to understand the data being transmitted –

First, let’s clean out the WH and State Department heads who both state “the buck stops here” – resign —

Next, in no way did we misconstrue the requests our military intelligence personnel were reporting in their frustration that requests for more security – denied….and in June, special troops brought home. Why, because this “Rev Wright-Obama pulpit” did not want to offend the local neighbors where he had pledged to remove oour presence and while all can talk including Madame Secretary of the DHS about data and the exabytes of information, code yellow is code yellow and w/drone overhead, the blood of our wonderful Ambassador and these other three Patriots standing all alone with such bravery to protect Life and ol ‘Glory, well, the blood of these fine gentlemen who Loved America so, so much is on the hands of Barry Obama and Hillary Clinton and Madame Secretary, as DHS Secretary, let’s Not worry about “the big data problem” which challenges your department and personnel, let’s worry that this calamity, this travesty, far bigger than even waterGate for no one died at Watergate, however this – breach in trust – treason – where data was real time and being passed to you and others at your rank where US security forces some 30 minutes away could have been on scene and our Ambassador and these three brave Americans were left to die – bottom line –

To the voters of Ohio, if in fact this election is supposedly to be determined by your vote, well, if you vote this administration back into office to further their dismissal of our beloved Republic and Constitution whether men or wormn, you, too have the blood of these four men on your hands as well – God is witness to all!

An exabyte problem which certainly exists when you Madame Secretary and Secretary of State and this “Chicago-Hollywood-Washington Express” dismissed any data coming over the wire clearly showing an attack and who was taking responsibility for such –

Again, my bytes will hopefully be clearly understood, “Treason I Shout and Shame on You I Say” – there is enough information now from breaking news and data which has now been received to move forward and call a trial for intentional acts upon the Constitution and the People of America to constitue – treason!

Once we can place politics aside and stand ready to protect fellow American soldier, someone’s son or daughter and when we see bytes of messages calling for increased security and an ongoing attack calling desperately for help and this WH and Secretary of State using their bytes to state these empty words, “the buck stops with me” – well it does and as a US Citizen who has called for a full “Benghazi Congressional Hearing” a few hours of such attack and using my bytes in calling for both these US officials to be held for suspicion of treason, let us remember that we will read exabytes from this biased media, however an oath of office is an oath of office and actions and bytes pointing to a flick few knew anything about as responsible for these deaths, well. before we worry about the exabytes Madame DHS Secretary, how about advising – We the People – for we abhor this intentional distraction at a time when our Ambassador was laying on the floor striggling to breathe from fire and smoke while You at the WH who leans to the “KGB Putinites” and whispers – watch with no care whatsoever! The blood of these fine men and their exabytes demanding security increase, all dismissed for the sole reason to appease your comrades on the streets….

You Sir at the WH and Madame at State, you have committed criminal action and no oneis above the law – the buck certianly does stop with you, now resign and stand in front of peers as We all seek to listen to every byte of the communications from the Ambassador the same day and the weeks prior seeking assistance…

There is no other issue more improtant to discuss than what has taken place at Benghazi and “Treason I Shout and Shame on You I Say” –

We are not fools here on “Main Street USA” and we will not be subject to your ongoing criticisms of America in front of your Muslim Brotherhood!

God Bless America!


Comment by Dan O'Connor

October 24, 2012 @ 12:05 am

These are not needles in a haystack but needles in needles. Big data and its analysis may hold many answers to health, economic, and homeland security issues. However, until that data can be tamed in some fashion it will exponentially grow.

And data creates data. Is a data black swan looming or is Phil correct for succintly stating ther have always been distractions? And what is our capacity to use this plethora if information?

Excellent questions and responses

Comment by HGRATTAN

October 24, 2012 @ 6:40 am

Dan and Phil,

Agreed: distractions have been and will be a part of our lives eternal.

Also agreed: big data may answer many questions.
Moreover, big data will raise as many more questions.
Howard Gardner’s Five Minds for the Future* argues and I agree that mental abilities (“minds”) to synthesize large swaths of data will be critical to success in the twenty-first century.

The synthesis-imperative will require fundamental and advanced critical thinking skills: inferences, recognition of assumptions, deductive reasoning, interpretation, and evaluation of arguments.**

This raises the question: why don’t more educators teach critical thinking skills as opposed to talking about why students don’t think critically.


**See: Think Watson: http://www.thinkwatson.com/assessments/watson-glaser

Comment by "Treason I Shout and Shame On You I Say" - Joe Citizen aka christopher tingus, cape cod

October 26, 2012 @ 1:14 pm

Let us continue to pray for the wonderful Amabassador and three Brave Patriots who were left behind – As I stood for 1,491 hours in three winters in silent protest on behalf of first responders and as I stood in silience front of the Cambride Police Department on behalf of police officers just doing their job to protest and this biased and divisive WH interfering in a local police matter in 9111 response w/a WH in its elitist and arrogant demeanor thinking that a beer on the WH lawn would somehow cover its direct and shameful interference in local law enforcement seeking to bring disharmony rather than staying out of matters which are not the WH busienss, shortly and now an elder, I will strive to come to Washington and carry my sign again in silience:

“Treason I Shout and Shame on You I Say” Joe Citizen

If I could make a citizen’s arrest of both Barry Obama and Hillary Clinton for suspicion of Treason and demanding a full “Benghazi Congressional Hearing” for the blood both Barry Obama and Hillary Clinton have on their hands, I would be at the foot of the Congressional stairs yesterday.

As a proud American citizen, so respectful of all law enforcement and military personnel everywhere and our intel folks who keep us safe day and night, I will strive to muster the strength to come to Washington and stand in silence as one citizen amongs millions and millions who feel that Barry Obama and Hillary Clinton – who I remind them that they are not above the law holding “entrusted” position – such malfeasance, such travesty and with laser beam on the weapons which were nbeing used to attack our beloved Patriots and our lock and key embassy, well, there is nothing more important to discuss for the blood of our beloved protectors are on the hands of the WH and State Department and anyone else who advised that those prepared to rally in response to pleas for help and were told to stand down, well….

“Treason I Shout and Shame on You I Say” – Joe Citizen

I appeal to those at DHS and other government agency personnel and to fellow citizens across our great Republic to get your signs printed – “Treason I Shout and Shame on You I Say” and stand in front of the Congressional stairs with me in honor of these heros, those that have given their precious Life for their country and for each of us so obligated to stand on behalf of these men and all the women and men in our Armed Services and folks such as at NSA and other who stand so willingly in their inherent Love of our great nation!

God Bless America!

Joe Citizen
aka Christopher Tingus
“Treason I Shout and Shame on You I Say” – christopher tingus, cape cod
“Main Street USA”
skype: christopher.tingus

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>