Andrew Leonard writes: For decades, so-called gift economy collaboration, in which the community as a whole benefits from the freely donated contributions of its members, has been a potent driver of Internet software evolution. As I wrote 16 years ago, when chronicling the birth of the Apache Web server, the success of open source software “testifies to the enduring vigor of the Internet’s cooperative, distributed approach to solving problems.” Hadoop [an open source platform which is becoming the standard platform for big data analytics], which down to its fundamental structural essence is a distributed approach to solving problems, emblematized this philosophy at its core.

So, in a sense, Hadoop’s success was just the same old story. But back in the mid-’90s, around the time that one of the first open source success stories, the Apache Web server, was taking off, I’m not sure that anyone would have predicted that the National Security Agency and CIA would end up becoming stalwart participants in the gift economy. Even though it makes total sense, in principle, that the fruits of government-funded software development should be shared with the general public, there’s still something cognitively disjunctive about intelligence agencies that shroud their every activity in great secrecy contributing to projects built on openness and transparency. On the one hand, employees of the NSA are appearing at conferences discussing how they have adapted Hadoop to solve the problems of dealing with unimaginably huge data sets, but on the other hand, we’re not supposed to know anything about what they are actually doing with that data.

The intertwining of the intelligence agencies with the larger open source software community could hardly be more incestuous. In 2008, a group of Yahoo employees that eventually included Doug Cutting formed a start-up designed to commercialize Hadoop called Cloudera. The CIA, through its In-Q-Tel (named after James Bond’s Q character) venture capital arm, was an early investor in, and customer of, Cloudera. The NSA built a significant piece of software that works “on top” of Hadoop called Accumulo designed to add sophisticated security controls managing how data could be accessed, and then promptly donated that code to the Apache Software Foundation. Later, a group of NSA software engineers formed another spinoff company, Sqrrl, to commercialize Accumulo.

What all this means is that the improvements to tools that the NSA is making, with the aim of more efficiently catching terrorists, are propagating into the private sector where they will be used by Facebook and Neftlix and Yahoo to more accurately target ads or influence our purchasing behavior or provide us with content algorithmically shaped to our very specific desires. And vice versa. Innovations and increased capabilities pioneered by private companies trickle back to the NSA. The collective boot-strapping never stops.

Again, in principle, there is nothing necessarily wrong going on here. There is no one to blame. Some of the fiercer apologists for unfettered free markets might complain that government involvement in open source projects unfairly competes with private sector proprietary businesses, but a much stronger case can be made that any software development work that is funded by taxpayer money should by definition be considered freely sharable with the wider public. The NSA should probably be applauded for helping to improve Hadoop. And if the capabilities unlocked by Hadoop result in the prevention of some horrific terrorist act, then every programmer who contributed a line of code to the project justly deserves some congratulation.

But there’s also an intriguing inversion occurring here of what, for better or worse, we might call the purpose of the Internet. The Internet was initially created by the U.S. government to facilitate the sharing of information between geographically separate research centers. The Internet took off in the mid-’90s in large part because the general public recognized it as a phenomenal tool for sharing information with each other. The fact that so much of the Internet’s infrastructure was also built from code that was freely shared seemed like a pleasing match of form and function.

Free software and open-source software evolution is frequently driven not so much by hope for financial gain but by individuals looking to solve their immediate engineering problems. Over time, on the Internet at large, one of those problems has turned out to be the gnarly challenge of how to manage all the data created by all those people sharing so promiscuously with each other. Hadoop can justly be seen as the natural response to all that promiscuous sharing. And it certainly helped solve the problems faced by engineers at Facebook and elsewhere.

But what ended up getting enabled by the success of Hadoop is something significantly different than good old peer-to-peer sharing. The ability to make sense out of petabytes of data isn’t necessarily useful to you or me. But it’s god’s gift to the profit-minded corporations and terrorist-seeking intelligence agencies seeking to leverage the data we generate for their own purposes, to measure our behavior and ultimately to influence it. That could mean Netflix figuring out exactly what combination of plot twists and acting talent proves irresistible to streaming video watchers or Facebook figuring out exactly how to stock our newsfeeds with advertisements that generate acceptable click-through or Twitter knowing exactly where we are on the surface of the planet so it can pop up a sponsored tweet pushing a coupon for a happy hour at the bar just down the street — or the NSA spotting a peculiar pattern of pressure cooker purchases. This is no longer about sharing information with each other; it’s about manipulation, control and punishment. It’s about keeping stock prices up. We’re a long, long way here from the ideal gift economy, where everyone brings their home-cooked delicacy to the potlatch. We’ve arrived at a destination where the tools offer more power to them than to us. [Continue reading…]