A cyborg who sees color through sound

Long before the term cyborg had been coined, Henry David Thoreau — who saw few if any advances in our inexorable movement away from our natural condition — declared: “men have become the tool of their tools.”

Creating the means for someone with total color blindness to be able to hear color, seems like an amazing idea, but we glimpse the dystopian potential of such technology when the beneficiary says the sounds coming from the strident colors of cleaning products displayed on the aisle of a supermarket are more enjoyable than the sound-color of the ocean.

Singularity Hub: What would your world be like if you couldn’t see color? For artist Neil Harbisson, a rare condition known as achromatopsia that made him completely color blind rendered that question meaningless. Not being able to see color at all meant that there was no blue in the sky or green in grass, and these descriptions were merely something to be taken on faith or memorized to get the correct answers in school.

But Neil’s life would change drastically when he met computer scientist Adam Montandon and with help from a few others, they developed the eyeborg, an electronic eye that transforms colors into sounds. Colors became meaningful for Neil in an experiential way, but one that was fundamentally different than how others described them.

This augmentation device wasn’t like a set of headphones that he could put on when he wanted to “listen” to the world around him, but became a permanent part of who he was. Though he had to memorize how the sounds corresponded to certain colors, in time the sounds became part of his perception and the way he “sees” the world. He even started to expand the range of what he could “see”, so that wavelengths of light outside of the visible range could be perceived.

In other words, he became cybernetic…

Neil boldly paints a picture of what the future holds where augmentation devices will alter how we experience the world. Whether for corrective or elective motives, people will someday adopt these technologies routinely, perhaps choosing artificial synesthesia as a means of seeing the world in a broader or deeper way.

In color theory, color is described in terms of three attributes: hue, saturation, and value. Hue is what we generally refer to with color terms — red, green, purple etc. Saturation is the intensity of a color — pale yellow, for instance, has less saturation than lemon yellow. And value refers to the lightness or darkness of a color. In moonlight, all we can perceive is color value, without any ability to register hues or saturation.

When color is understood in these terms, achromatopsia does not have to be viewed as a lack of color vision since, at least in some cases, it can actually lead to the experience of a refinement of sight.

As much as Neil Harbisson might feel that technology has enhanced his perception of the world, I find it depressing that anyone would fail to see that in order to value this kind of nominal extension of the senses requires first that we underestimate the subtlety of human perception.

In An Anthropologist On Mars, Oliver Sacks describes the experience of Jonathan I., a 65-year old artist who suddenly lost his color vision as a result of concussion sustained in a car accident.

As an artist, the loss was devastating.

He knew the colors of everything, with an extraordinary exactness (he could give not only the names but the numbers of colors as these were listed in a Pantone chart of hues he had used for many years). He could identify the green of van Gogh’s billiard table in this way unhesitatingly. He knew all the colors in his favorite paintings, but could no longer see them, either when he looked or in his mind’s eye…

As the months went by, he particularly missed the brilliant colors of spring — he had always loved flowers, but now he could only distinguish them by shape or smell. The blue jays were brilliant no longer, — their blue, curiously, was now seen as pale grey. He could no longer see the clouds in the sky, their whiteness, or off-whiteness as he saw them, being scarcely distinguishable from the azure, which seemed bleached to a pale grey…

His initial sense of helplessness started to give way to a sense of resolution — he would paint in black and white, if he could not paint in color; he would try to live in a black-and-white world as fully as he could. This resolution was strengthened by a singular experience, about five weeks after his accident, as he was driving to the studio one morning. He saw the sunrise over the highway, the blazing reds all turned into black: “The sun rose like a bomb, like some enormous nuclear explosion”, he said later. “Had anyone ever seen a sunrise in this way before?”

Inspired by the sunrise, he started painting again—he started, indeed, with a black-and-white painting that he called Nuclear Sunrise, and then went on to the abstracts he favored, but now painting in black and white only. The fear of blindness continued to haunt him but, creatively transmuted, shaped the first “real” paintings he did after his color experiments. Black-and-white paintings he now found he could do, and do very well. He found his only solace working in the studio, and he worked fifteen, even eighteen, hours a day. This meant for him a kind of artistic survival: “I felt if I couldn’t go on painting”, he said later, “I wouldn’t want to go on at all.”…

Color perception had been an essential part not only of Mr. I.’s visual sense, but his aesthetic sense, his sensibility, his creative identity, an essential part of the way he constructed his world — and now color was gone, not only in perception, but in imagination and memory as well. The resonances of this were very deep. At first he was intensely, furiously conscious of what he had lost (though “conscious”, so to speak, in the manner of an amnesiac). He would glare at an orange in a state of rage, trying to force it to resume its true color. He would sit for hours before his (to him) dark grey lawn, trying to see it, to imagine it, to remember it, as green. He found himself now not only in an impoverished world, but in an alien, incoherent, and almost nightmarish one. He expressed this soon after his injury, better than he could in words, in some of his early, desperate paintings.

But then, with the “apocalyptic” sunrise, and his painting of this, came the first hint of a change, an impulse to construct the world anew, to construct his own sensibility and identity anew. Some of this was conscious and deliberate: retraining his eyes (and hands) to operate, as he had in his first days as an artist. But much occurred below this level, at a level of neural processing not directly accessible to consciousness or control. In this sense, he started to be redefined by what had happened to him — redefined physiologically, psychologically, aesthetically — and with this there came a transformation of values, so that the total otherness, the alienness of his V1 world, which at first had such a quality of horror and nightmare, came to take on, for him, a strange fascination and beauty…

At once forgetting and turning away from color, turning away from the chromatic orientation and habits and strategies of his previous life, Mr. I., in the second year after his injury, found that he saw best in subdued light or twilight, and not in the full glare of day. Very bright light tended to dazzle and temporarily blind him — another sign of damage to his visual systems—but he found the night and nightlife peculiarly congenial, for they seemed to be “designed”, as he once said, “in terms of black and white.”

He started becoming a “night person”, in his own words, and took to exploring other cities, other places, but only at night. He would drive, at random, to Boston or Baltimore, or to small towns and villages, arriving at dusk, and then wandering about the streets for half the night, occasionally talking to a fellow walker, occasionally going into little diners: “Everything in diners is different at night, at least if it has windows. The darkness comes into the place, and no amount of light can change it. They are transformed into night places. I love the nighttime”, Mr. I. said. “Gradually I am becoming a night person. It’s a different world: there’s a lot of space — you’re not hemmed in by streets, by people — It’s a whole new world.”…

Most interesting of all, the sense of profound loss, and the sense of unpleasantness and abnormality, so severe in the first months following his head injury, seemed to disappear, or even reverse. Although Mr. I. does not deny his loss, and at some level still mourns it, he has come to feel that his vision has become “highly refined”, “privileged”, that he sees a world of pure form, uncluttered by color. Subtle textures and patterns, normally obscured for the rest of us because of their embedding in color, now stand out for him…

He feels he has been given “a whole new world”, which the rest of us, distracted by color, are insensitive to. He no longer thinks of color, pines for it, grieves its loss. He has almost come to see his achromatopsia as a strange gift, one that has ushered him into a new state of sensibility and being.

Perhaps the greatest ability we are endowed with by nature resides in none of our individual senses but in our surprising powers of adaptation.

In the current technology-worshiping milieu we are indeed becoming the tools of our tools, but in a more literal sense than Thoreau might have imagined. Unwitting slaves, chained to machines — through devices that supposedly form indispensable connections to the world we are gradually becoming disconnected from what it means to be human.

Print Friendly, PDF & Email
Facebooktwittermail