John Lanchester writes: In August, the editor of the Guardian rang me up and asked if I would spend a week in New York, reading the GCHQ files whose UK copy the Guardian was forced to destroy. His suggestion was that it might be worthwhile to look at the material not from a perspective of making news but from that of a novelist with an interest in the way we live now.

I took Alan Rusbridger up on his invitation, after an initial reluctance that was based on two main reasons. The first of them was that I don’t share the instinctive sense felt by many on the left that it is always wrong for states to have secrets. I’d put it more strongly than that: democratic states need spies.

The philosopher Karl Popper, observing the second world war from his academic post in New Zealand, came up with a great title for his major work of political thought: The Open Society and Its Enemies. It is, in its way, a shocking phrase – why would the open society have enemies? (But then, the title of Charles Repington’s The First World War, published in 1920, was shocking too, because it implied that there would be another one.)

We do have enemies, though, enemies who are in deadly earnest; enemies who wish you reading this dead, whoever you are, for no other reason than that you belong to a society like this one. We have enemies who are seeking to break into our governments’ computers, with the potential to destroy our infrastructure and, literally, make the lights go out; we have enemies who want to kill as many of us, the more innocent the better, as possible, by any means possible, as a deliberate strategy; we have enemies who want to develop nuclear weapons, and thereby vastly raise the stakes for international diplomacy and the threat of terrorism; and we have common-or-garden serious criminals, who also need watching and catching.

I get all that. It doesn’t thrill me to bits that the state has to use the tools of electronic surveillance to keep us safe, but it seems clear to me that it does, and that our right to privacy needs to be qualified, just as our other rights are qualified, in the interest of general security and the common good.

My week spent reading things that were never meant to be read by outsiders was, from this point of view, largely reassuring. Most of what GCHQ does is exactly the kind of thing we all want it to do. It takes an interest in places such as the Horn of Africa, Iran, and North Korea; it takes an interest in energy security, nuclear proliferation, and in state-sponsored computer hacking.

There doesn’t seem to be much in the documents about serious crime, for which GCHQ has a surveillance mandate, but it seems that much of this activity is covered by warrants that belong to other branches of the security apparatus. Most of this surveillance is individually targeted: it concerns specific individuals and specific acts (or intentions to act), and as such, it is not the threat.

Even Julian Assange thinks that, and said as much in his alarming and perceptive book Cypherpunks: “Individual targeting is not the threat.” When the state has specific enemies and knows who they are and the kind of harm they intend, it is welcome to target them to make the rest of our polity safe. I say again, on the evidence I’ve seen, this is mainly what GCHQ does. I would add that the Guardian and its partners have gone to a lot of trouble to prevent any unnecessarily damaging detail about this work being published.

The problems with GCHQ are to be found in the margins of the material – though they are at the centre of the revelations that have been extracted from the Snowden disclosures, and with good reason. The problem and the risk comes in the area of mass capture of data, or strategic surveillance. This is the kind of intelligence gathering that sucks in data from everyone, everywhere: from phones, internet use from email to website visits, social networking, instant messaging and video calls, and even areas such as video gaming; in short, everything digital.

In the US, the Prism programme may have given the NSA access to the servers of companies such as Google and Facebook; in the UK, GCHQ has gained a similar degree of access via its Tempora programme, and the two of them together have a cable- and network-tapping capabilities collectively called Upstream, which have the ability to intercept anything that travels over the internet. This data is fed into a database called XKeyscore, which allows analysts to extract information “in real time”, ie immediately, from a gigantic amount of hoovered-up data.

In addition, the NSA has encouraged technology companies to install secret weaknesses or “backdoors” into their commercially available, supposedly secure products. They have spent a very great deal of money ($250m a year alone on weakening encryption), on breaking commercially available security products. Other revelations have been published in Der Spiegel, and concern the NSA exploitation of technology such as the iPhone.

What this adds up to is a new thing in human history: with a couple of clicks of a mouse, an agent of the state can target your home phone, or your mobile, or your email, or your passport number, or any of your credit card numbers, or your address, or any of your log-ins to a web service.

Using that “selector”, the state can get access to all the content of your communications, via any of those channels; can gather information about anyone you communicate with, can get a full picture of all your internet use, can track your location online and offline. It can, in essence, know everything about you, including – thanks to the ability to look at your internet searches – what’s on your mind.

To get a rough version of this knowledge, a state once had to bug phones manually, break into houses and intercept letters, and deploy teams of trained watchers to follow your whereabouts. Even then it was a rough and approximate process, vulnerable to all sorts of human error and countermeasures. It can now have something much better than that, a historically unprecedented panoply of surveillance, which it can deploy in a matter of seconds.

This process is not without supervision, of course. In order to target you via one of these “selectors” – that’s the technical term – the agent of the state will have to type into a box on his or her computer screen a Miranda number, to show that the process is taking place in response to a specific request for information, and will also need to select a justification under the Human Rights Act. That last isn’t too arduous, because the agent can choose the justification from a drop-down menu. This is the way we live now.

And yet nobody, at least in Britain, seems to care. In the UK there has been an extraordinary disconnect between the scale and seriousness of what Snowden has revealed, and the scale and seriousness of the response. One of the main reasons for that, I think, is that while some countries are interested in rights, in Britain we are more focused on wrongs. [Continue reading…]