How big is the NSA’s haystack?

Sounding like a logic-defying Zen master, NSA chief Keith Alexander this week claimed: “You need a haystack to find a needle.” I pity any foreign journalist who took on the challenge of attempting to translate that into an intelligible statement.

Alexander’s staff, on the other hand, seem less inclined to speak in riddles — though just as eager to portray the operations of a police state in terms more appropriate for a kindergarten.

Anyone who might have previously been wondering whether the NSA is really watching everyone, should no longer be in any doubt — once they’ve grasped the humans-like-rabbits metaphor that we each stand a certain number of “hops” apart.

Following National Security Agency Deputy Director Chris Inglis’s revelation that the agency conducts surveillance as many as “three hops” away from a suspected terrorist, Sean Gallagher digs into the numbers:

A great deal of research has been done into the interconnectedness of people in the Internet age. Social scientists, mathematicians, and computer scientists have explored the “small world” phenomenon with studies and experiments for over 50 years, and their findings show that the “small world” keeps getting smaller as technology advances. In 1979, chair and founder of MIT’s political science department Ithiel de Sola Pool and the University of Michigan’s Manfred Kochen published a paper titled “Contacts and Influence,” which draws on a decade of research into social networks. De Sola Pool and Kochen posited that “in a country the size of the United States, if acquaintanceship were random and the mean acquaintance volume were 1,000, the mean length of minimum chain between pairs of persons would be well under two intermediaries.”

In other words, if the average person in the US has contact with and is acquainted with 1,000 others (through brief interactions, such as an e-mail or a phone call, or through stronger associations), then we’re at most two hops from anyone else in the US. Ergo, if any one person in the US is one hop from a terrorist, chances are good that you are three hops away.

The actual degrees of separation between people may be somewhat larger because the population of the US has grown significantly since 1979; our interconnectedness with the world at large has grown as well, widening the potential links between people. Live in a major metropolitan center in the US and you’re bound to be two degrees of separation away from someone in a country that’s of interest to the NSA. For example: I have been a regular customer of restaurants owned by Baltimore’s Karzai family, which is headed by a brother of Afghan President Hamid Karzai—two hops. I’m also, according to LinkedIn, two degrees of separation away from President Obama. Am I a good guy or a bad guy?

The Internet has blown the level of interconnectedness though the proverbial roof—we now have e-mail, social media, and instant message interactions with people we’ll never meet in real life and in places we’ll never go. A 2007 study by Carnegie Mellon University machine learning researcher Jure Leskovec and Microsoft Research’s Eric Horvitz found that the average number of hops between any two arbitrary Microsoft Messenger users, based on interaction, was 6.6. And a study of Twitter feeds published in 2011 found the average degree of separation between random Twitter users to be only 3.43.

So even if the NSA limited its surveillance activities—and by “surveillance” I mean active probing of the content of communications of an individual—to people within two hops of suspected terrorists, that’s a sizable population. Three ratchets it up to hundreds of millions or potentially billions of people, especially when the definition of a hop is based on relationships so casual we could create them by accidentally clicking on a link in a spam e-mail. So far, we know that there have been about 20,000 requests for FISA warrants to surveil domestic targets since 2001, but if those warrants covered three hops from the suspects at the center of the requests—depending on how tightly or loosely the NSA defines a relationship—three hops could encompass as much as 50 percent of the Internet-using population of the world. [Continue reading…]

Print Friendly, PDF & Email