Shane Harris writes: A decade ago, a Pentagon research project called “Total Information Awareness” sparked a mass panic because of its seemingly Orwellian interest in categorizing and mining every aspect of our digital lives. It was “the supersnoop’s dream,” declared William Safire of the New York Times, a “computerized dossier on your private life from commercial sources, [combined with] every piece of information that government has about you….”

If this sounds reminiscent of the current uproar over NSA surveillance, you’re paying attention. That’s because the NSA monitoring tools are very similar to — and, in many cases are directly based on — the technology that Total Information Awareness (TIA) tried to use.

The story of that convergence starts on the morning of Feb. 2, 2002, when retired Admiral John Poindexter drove to the headquarters of the National Security Agency at Ft. Meade, Maryland, and sat down with the agency’s deputy director, an NSA veteran named Bill Black. Poindexter, a former White House national security adviser, was now running the TIA program at the Defense Advanced Research Projects Agency, the organization that tackles some of the hardest engineering and technology challenges in the Pentagon. Poindexter thought TIA was an innovative new way to stop terrorist attacks, and he wanted the NSA to help him test it.

The idea, he explained to Black, was to give U.S. intelligence analysts access to the vast universe of electronic information stored in private databases that might be useful for detecting the next plot. Data such as phone call records, emails, and Internet searches. Poindexter wanted to build what he called a “system of systems” that would access all this raw information, sort and analyze it, and hopefully find indications of terrorist plotting.

The NSA was the biggest collector of electronic data in the government, and Poindexter thought the NSA would be a natural partner in his endeavor. But what he didn’t know was that under secret orders from President George W. Bush, the NSA was already building its own version of Total Information Awareness. Fewer than 100 people at the NSA knew that for the past few months, the agency had been monitoring the phone calls and other electronic communications of Americans, and that it was obtaining copies of domestic phone call records and looking at them for potential clues about terrorist attacks.

Poindexter left Ft. Meade that day with no firm commitment from Black that the NSA would assist in his research. And TIA didn’t last long. Although Poindexter’s work wasn’t classified, the press soon caught wind of his grand data-mining ambitions, and Poindexter was held up as the poster boy for intrusive government surveillance. “I think it’s fair to say that in the country’s history there has never been proposed a program with something this far reaching in terms of surveillance capacity,” said Sen. Ron Wyden at the time. “And my sense is that the country just does not want to unleash a bunch of virtual bloodhounds to go sniffing into the medical, financial and travel records of law-abiding Americans.”

TIA was officially shut down in 2003, and Poindexter left the government. But this wasn’t the end of his grand vision.

In a secret negotiation, members of Congress, some of whom had been among Poindexter’s critics, reached an agreement to keep TIA research going, and to fund it from the classified portion of the military budget, the so-called “black budget.” TIA’s research components were given new cover names, and the program was moved under the control of the very agency that Poindexter had originally wanted to help him — the NSA. There, Poindexter’s ideas were incorporated into NSA’s surveillance activities, the latest glimpses of which we have seen in the past two weeks. [Continue reading…]