Lesson 7 - Why Does the Future Need Us PDF

+-----------------------------------------------------------------------+ | ***Course Code and Title: GE 106* SCIENCE, TECHNOLOGY and SOCIETY** | +=======================================================================+ | **Lesson 7 Week 12** | +---...

+-----------------------------------------------------------------------+ | ***Course Code and Title: GE 106* SCIENCE, TECHNOLOGY and SOCIETY** | +=======================================================================+ | **Lesson 7 Week 12** | +-----------------------------------------------------------------------+ | **Topic:** Why does the future not need us? | +-----------------------------------------------------------------------+ | **Introduction:** | | | | **Let us try to reflect this remarkable passage:** | | | | **"Our most powerful 21^st^- century technologies- robotics, genetic | | engineering and nanotech- are threatening to make humans an | | endangered species."** | | | | - **Bill Joy** | +-----------------------------------------------------------------------+ | **Learning Objectives:** | +-----------------------------------------------------------------------+ | Content: | | | | **Why the Future Doesn\'t Need Us** | | | | *Bill Joy (April 2000)* | | | | From the moment I became involved in the creation of new | | technologies, their ethical dimensions have concerned me, but it was | | only in the autumn of 1998 that I became anxiously aware of how great | | are the dangers facing us in the 21st century. I can date the onset | | of my unease to the day I met Ray Kurzweil, the deservedly famous | | inventor of the first reading machine for the blind and many other | | amazing things. | | | | Ray and I were both speakers at George Gilder\'s Telecoms conference, | | and I encountered him by chance in the bar of the hotel after both | | our sessions were over. I was sitting with John Searle, a Berkeley | | philosopher who studies consciousness. While we were talking, Ray | | approached and a conversation began, the subject of which haunts me | | to this day. | | | | I had missed Ray\'s talk and the subsequent panel that Ray and John | | had been on, and they now picked right up where they\'d left off, | | with Ray saying that the rate of improvement of technology was going | | to accelerate and that we were going to become robots or fuse with | | robots or something like that, and John countering that this | | couldn\'t happen, because the robots couldn\'t be conscious. | | | | While I had heard such talk before, I had always felt sentient robots | | were in the realm of science fiction. But now, from someone I | | respected, I was hearing a strong argument that they were a near-term | | possibility. I was taken aback, especially given Ray\'s proven | | ability to imagine and create the future. I already knew that new | | technologies like genetic engineering and nanotechnology were giving | | us the power to remake the world, but a realistic and imminent | | scenario for intelligent robots surprised me. | | | | It is easy to get jaded about such breakthroughs. We hear in the news | | almost every day of technological or scientific advance. Yet this was | | no ordinary prediction. In the hotel bar, Ray gave me a partial | | preprint of his then-forthcoming book *The Age of Spiritual | | Machines,* which outlined a utopia he foresaw---one in which humans | | gained near immortality by becoming one with robotic technology. On | | reading it, my sense of unease only intensified; I felt sure he had | | to be understating the dangers, understating the probability of a bad | | outcome along this path. | | | | I found myself most troubled by a passage detailing a *dys*topian | | scenario: | | | | **THE NEW LUDDITE CHALLENGE** | | | | First let us postulate that the computer scientists succeed in | | developing intelligent machines that can do all things better than | | human beings can do them. In that case presumably all work will be | | done by vast, highly organized systems of machines and no human | | effort will be necessary. Either of two cases might occur. The | | machines might be permitted to make all of their own decisions | | without human oversight, or else human control over the machines | | might be retained. | | | | If the machines are permitted to make all their own decisions, we | | can\'t make any conjectures as to the results, because it is | | impossible to guess how such machines might behave. We only point out | | that the fate of humans would be at the mercy of the machines. It | | might be argued that humanity would never be foolish enough to hand | | over all the power to the machines. But we are suggesting neither | | that people would voluntarily turn power over to the machines nor | | that the machines would willfully seize power. What we do suggest is | | that the human race might easily permit itself to drift into a | | position of such dependence on the machines that it would have no | | practical choice but to accept all of the machines\' decisions. As | | society and the problems that face it become more and more complex | | and machines become more and more intelligent, people will let | | machines make more of their decisions for them, simply because | | machine-made decisions will bring better results than man-made ones. | | Eventually a stage may be reached at which the decisions necessary to | | keep the system running will be so complex that human beings will be | | incapable of making them intelligently. At that stage, the machines | | will be in effective control. People won\'t be able to just turn the | | machines off, because they will be so dependent on them that turning | | them off would amount to suicide. | | | | On the other hand, it is possible that human control over the | | machines may be retained. In that case the average man may have | | control over certain private machines of his own, such as his car or | | his personal computer, but control over large systems of machines | | will be in the hands of a tiny elite---just as it is today, but with | | two differences. Due to improved techniques the elite will have | | greater control over the masses; and because human work will no | | longer be necessary the masses will be superfluous, a useless burden | | on the system. If the elite is ruthless, they may simply decide to | | exterminate the mass of humanity. If they are humane, they may use | | propaganda or other psychological or biological techniques to reduce | | the birth rate until the mass of humanity becomes extinct, leaving | | the world to the elite. Or, if the elite consists of soft-hearted | | liberals, they may decide to play the role of good shepherds to the | | rest of the human race. They will see to it that everyone\'s physical | | needs are satisfied, that all children are raised under | | psychologically hygienic conditions, that everyone has a wholesome | | hobby to keep him busy, and that anyone who may become dissatisfied | | undergoes \"treatment\" to cure his \"problem.\" Of course, life will | | be so purposeless that people will have to be biologically or | | psychologically engineered either to remove their need for the power | | process or make them \"sublimate\" their drive for power into some | | harmless hobby. These engineered human beings may be happy in such a | | society, but they will most certainly not be free. They will have | | been reduced to the status of domestic animals.^1^ | | | | In the book, you don\'t discover until you turn the page that the | | author of this passage is Theodore Kaczynski---the Unabomber. I am no | | apologist for Kaczynski. His bombs killed three people during a | | 17-year terror campaign and wounded many others. One of his bombs | | gravely injured my friend David Gelernter, one of the most brilliant | | and visionary computer scientists of our time. Like many of my | | colleagues, I felt that I could easily have been the Unabomber\'s | | next target. | | | | Kaczynski\'s actions were murderous and, in my view, criminally | | insane. He is clearly a Luddite, but simply saying this does not | | dismiss his argument; as difficult as it is for me to acknowledge, I | | saw some merit in the reasoning in this single passage. I felt | | compelled to confront it. | | | | Kaczynski\'s dystopian vision describes unintended consequences, a | | well-known problem with the design and use of technology, and one | | that is clearly related to Murphy\'s law---\"Anything that can go | | wrong, will.\" (Actually, this is Finagle\'s law, which in itself | | shows that Finagle was right.) Our overuse of antibiotics has led to | | what may be the biggest such problem so far: the emergence of | | antibiotic-resistant and much more dangerous bacteria. Similar things | | happened when attempts to eliminate malarial mosquitoes using DDT | | caused them to acquire DDT resistance; malarial parasites likewise | | acquired multi-drug-resistant genes.^2^ | | | | The cause of many such surprises seems clear: The systems involved | | are complex, involving interaction among and feedback between many | | parts. Any changes to such a system will cascade in ways that are | | difficult to predict; this is especially true when human actions are | | involved. | | | | I started showing friends the Kaczynski quote from *The Age of | | Spiritual Machines;* I would hand them Kurzweil\'s book, let them | | read the quote, and then watch their reaction as they discovered who | | had written it. At around the same time, I found Hans Moravec\'s book | | *Robot: Mere Machine to Transcendent Mind.* Moravec is one of the | | leaders in robotics research, and was a founder of the world\'s | | largest robotics research program, at Carnegie Mellon University. | | *Robot* gave me more material to try out on my friends---material | | surprisingly supportive of Kaczynski\'s argument. For example: | | | | **The Short Run (Early 2000s)** | | | | Biological species almost never survive encounters with superior | | competitors. Ten million years ago, South and North America were | | separated by a sunken Panama isthmus. South America, like Australia | | today, was populated by marsupial mammals, including pouched | | equivalents of rats, deers, and tigers. When the isthmus connecting | | North and South America rose, it took only a few thousand years for | | the northern placental species, with slightly more effective | | metabolisms and reproductive and nervous systems, to displace and | | eliminate almost all the southern marsupials. | | | | In a completely free marketplace, superior robots would surely affect | | humans as North American placentals affected South American | | marsupials (and as humans have affected countless species). Robotic | | industries would compete vigorously among themselves for matter, | | energy, and space, incidentally, driving their price beyond human | | reach. Unable to afford the necessities of life, biological humans | | would be squeezed out of existence. | | | | There is probably some breathing room because we do not live in a | | completely free marketplace. Government coerces nonmarket behavior, | | especially by collecting taxes. Judiciously applied, governmental | | coercion could support human populations in high style on the fruits | | of robot labor, perhaps for a long while. | | | | A textbook dystopia---and Moravec is just getting wound up. He goes | | on to discuss how our main job in the 21st century will be \"ensuring | | continued cooperation from the robot industries\" by passing laws | | decreeing that they be \"nice,\" and to describe how seriously | | dangerous a human can be \"once transformed into an unbounded | | superintelligent robot.\"^3^ Moravec\'s view is that the robots will | | eventually succeed us---that humans clearly face extinction. | | | | I decided it was time to talk to my friend Danny Hillis. Danny became | | famous as the cofounder of Thinking Machines Corporation, which built | | a very powerful parallel supercomputer. Despite my current job title | | of Chief Scientist at Sun Microsystems, I am more a computer | | architect than a scientist, and I respect Danny\'s knowledge of the | | information and physical sciences more than that of any other single | | person I know. Danny is also a highly regarded futurist who thinks | | long-term---four years ago he started the Long Now Foundation, which | | is building a clock designed to last 10,000 years, in an attempt to | | draw attention to the pitifully short attention span of our society. | | (See \"[[Test of | | Time]](https://www.wired.com/2000/03/test-of-time/),\"*Wi | | red* | | 8.03, page 78.) | | | | So I flew to Los Angeles for the express purpose of having dinner | | with Danny and his wife, Pati. I went through my now-familiar | | routine, trotting out the ideas and passages that I found so | | disturbing. Danny\'s answer---directed specifically at Kurzweil\'s | | scenario of humans merging with robots---came swiftly, and quite | | surprised me. He said, simply, that the changes would come gradually, | | and that we would get used to them. | | | | But I guess I wasn\'t totally surprised. I had seen a quote from | | Danny in Kurzweil\'s book in which he said, \"I\'m as fond of my body | | as anyone, but if I can be 200 with a body of silicon, I\'ll take | | it.\" It seemed that he was at peace with this process and its | | attendant risks, while I was not. | | | | While talking and thinking about Kurzweil, Kaczynski, and Moravec, I | | suddenly remembered a novel I had read almost 20 years ago -*The | | White Plague,* by Frank Herbert---in which a molecular biologist is | | driven insane by the senseless murder of his family. To seek revenge | | he constructs and disseminates a new and highly contagious plague | | that kills widely but selectively. (We\'re lucky Kaczynski was a | | mathematician, not a molecular biologist.) I was also reminded of the | | Borg of*Star Trek,* a hive of partly biological, partly robotic | | creatures with a strong destructive streak. Borg-like disasters are a | | staple of science fiction, so why hadn\'t I been more concerned about | | such robotic dystopias earlier? Why weren\'t other people more | | concerned about these nightmarish scenarios? | | | | Part of the answer certainly lies in our attitude toward the new---in | | our bias toward instant familiarity and unquestioning acceptance. | | Accustomed to living with almost routine scientific breakthroughs, we | | have yet to come to terms with the fact that the most compelling | | 21st-century technologies---robotics, genetic engineering, and | | nanotechnology---pose a different threat than the technologies that | | have come before. Specifically, robots, engineered organisms, and | | nanobots share a dangerous amplifying factor: They can | | self-replicate. A bomb is blown up only once---but one bot can become | | many, and quickly get out of control. | | | | Much of my work over the past 25 years has been on computer | | networking, where the sending and receiving of messages creates the | | opportunity for out-of-control replication. But while replication in | | a computer or a computer network can be a nuisance, at worst it | | disables a machine or takes down a network or network service. | | Uncontrolled self-replication in these newer technologies runs a much | | greater risk: a risk of substantial damage in the physical world. | | | | Each of these technologies also offers untold promise: The vision of | | near immortality that Kurzweil sees in his robot dreams drives us | | forward; genetic engineering may soon provide treatments, if not | | outright cures, for most diseases; and nanotechnology and | | nanomedicine can address yet more ills. Together they could | | significantly extend our average life span and improve the quality of | | our lives. Yet, with each of these technologies, a sequence of small, | | individually sensible advances leads to an accumulation of great | | power and, concomitantly, great danger. | | | | What was different in the 20th century? Certainly, the technologies | | underlying the weapons of mass destruction (WMD)---nuclear, | | biological, and chemical (NBC)---were powerful, and the weapons an | | enormous threat. But building nuclear weapons required, at least for | | a time, access to both rare---indeed, effectively unavailable---raw | | materials and highly protected information; biological and chemical | | weapons programs also tended to require large-scale activities. | | | | The 21st-century technologies---genetics, nanotechnology, and | | robotics (GNR)---are so powerful that they can spawn whole new | | classes of accidents and abuses. Most dangerously, for the first | | time, these accidents and abuses are widely within the reach of | | individuals or small groups. They will not require large facilities | | or rare raw materials. Knowledge alone will enable the use of them. | | | | Thus, we have the possibility not just of weapons of mass destruction | | but of knowledge-enabled mass destruction (KMD), this destructiveness | | hugely amplified by the power of self-replication. | | | | I think it is no exaggeration to say we are on the cusp of the | | further perfection of extreme evil, an evil whose possibility spreads | | well beyond that which weapons of mass destruction bequeathed to the | | nation-states, on to a surprising and terrible empowerment of extreme | | individuals. | | | | Nothing about the way I got involved with computers suggested to me | | that I was going to be facing these kinds of issues. | | | | My life has been driven by a deep need to ask questions and find | | answers. When I was 3, I was already reading, so my father took me to | | the elementary school, where I sat on the principal\'s lap and read | | him a story. I started school early, later skipped a grade, and | | escaped into books---I was incredibly motivated to learn. I asked | | lots of questions, often driving adults to distraction. | | | | As a teenager I was very interested in science and technology. I | | wanted to be a ham radio operator but didn\'t have the money to buy | | the equipment. Ham radio was the Internet of its time: very | | addictive, and quite solitary. Money issues aside, my mother put her | | foot down---I was not to be a ham; I was antisocial enough already. | | | | I may not have had many close friends, but I was awash in ideas. By | | high school, I had discovered the great science fiction writers. I | | remember especially Heinlein\'s *Have Spacesuit Will Travel* and | | Asimov\'s *I, Robot,* with its Three Laws of Robotics. I was | | enchanted by the descriptions of space travel, and wanted to have a | | telescope to look at the stars; since I had no money to buy or make | | one, I checked books on telescope-making out of the library and read | | about making them instead. I soared in my imagination. | | | | Thursday nights my parents went bowling, and we kids stayed home | | alone. It was the night of Gene Roddenberry\'s original *Star Trek,* | | and the program made a big impression on me. I came to accept its | | notion that humans had a future in space, Western-style, with big | | heroes and adventures. Roddenberry\'s vision of the centuries to come | | was one with strong moral values, embodied in codes like the Prime | | Directive: to not interfere in the development of less | | technologically advanced civilizations. This had an incredible appeal | | to me; ethical humans, not robots, dominated this future, and I took | | Roddenberry\'s dream as part of my own. | | | | I excelled in mathematics in high school, and when I went to the | | University of Michigan as an undergraduate engineering student I took | | the advanced curriculum of the mathematics majors. Solving math | | problems was an exciting challenge, but when I discovered computers I | | found something much more interesting: a machine into which you could | | put a program that attempted to solve a problem, after which the | | machine quickly checked the solution. The computer had a clear notion | | of correct and incorrect, true and false. Were my ideas correct? The | | machine could tell me. This was very seductive. | | | | I was lucky enough to get a job programming early supercomputers and | | discovered the amazing power of large machines to numerically | | simulate advanced designs. When I went to graduate school at UC | | Berkeley in the mid-1970s, I started staying up late, often all | | night, inventing new worlds inside the machines. Solving problems. | | Writing the code that argued so strongly to be written. | | | | In *The Agony and the Ecstasy,* Irving Stone\'s biographical novel of | | Michelangelo, Stone described vividly how Michelangelo released the | | statues from the stone, \"breaking the marble spell,\" carving from | | the images in his mind.^4^ In my most ecstatic moments, the software | | in the computer emerged in the same way. Once I had imagined it in my | | mind I felt that it was already there in the machine, waiting to be | | released. Staying up all night seemed a small price to pay to free | | it---to give the ideas concrete form. | | | | After a few years at Berkeley I started to send out some of the | | software I had written---an instructional Pascal system, Unix | | utilities, and a text editor called vi (which is still, to my | | surprise, widely used more than 20 years later)---to others who had | | similar small PDP-11 and VAX minicomputers. These adventures in | | software eventually turned into the Berkeley version of the Unix | | operating system, which became a personal \"success disaster\"---so | | many people wanted it that I never finished my PhD. Instead I got a | | job working for Darpa putting Berkeley Unix on the Internet and | | fixing it to be reliable and to run large research applications well. | | This was all great fun and very rewarding. And, frankly, I saw no | | robots here, or anywhere near. | | | | Still, by the early 1980s, I was drowning. The Unix releases were | | very successful, and my little project of one soon had money and some | | staff, but the problem at Berkeley was always office space rather | | than money---there wasn\'t room for the help the project needed, so | | when the other founders of Sun Microsystems showed up I jumped at the | | chance to join them. At Sun, the long hours continued into the early | | days of workstations and personal computers, and I have enjoyed | | participating in the creation of advanced microprocessor technologies | | and Internet technologies such as Java and Jini. | | | | From all this, I trust it is clear that I am not a Luddite. I have | | always, rather, had a strong belief in the value of the scientific | | search for truth and in the ability of great engineering to bring | | material progress. The Industrial Revolution has immeasurably | | improved everyone\'s life over the last couple hundred years, and I | | always expected my career to involve the building of worthwhile | | solutions to real problems, one problem at a time. | | | | I have not been disappointed. My work has had more impact than I had | | ever hoped for and has been more widely used than I could have | | reasonably expected. I have spent the last 20 years still trying to | | figure out how to make computers as reliable as I want them to be | | (they are not nearly there yet) and how to make them simple to use (a | | goal that has met with even less relative success). Despite some | | progress, the problems that remain seem even more daunting. | | | | But while I was aware of the moral dilemmas surrounding technology\'s | | consequences in fields like weapons research, I did not expect that I | | would confront such issues in my own field, or at least not so soon. | | | | Perhaps it is always hard to see the bigger impact while you are in | | the vortex of a change. Failing to understand the consequences of our | | inventions while we are in the rapture of discovery and innovation | | seems to be a common fault of scientists and technologists; we have | | long been driven by the overarching desire to know that is the nature | | of science\'s quest, not stopping to notice that the progress to | | newer and more powerful technologies can take on a life of its own. | | | | I have long realized that the big advances in information technology | | come not from the work of computer scientists, computer architects, | | or electrical engineers, but from that of physical scientists. The | | physicists Stephen Wolfram and Brosl Hasslacher introduced me, in the | | early 1980s, to chaos theory and nonlinear systems. In the 1990s, I | | learned about complex systems from conversations with Danny Hillis, | | the biologist Stuart Kauffman, the Nobel-laureate physicist Murray | | Gell-Mann, and others. Most recently, Hasslacher and the electrical | | engineer and device physicist Mark Reed have been giving me insight | | into the incredible possibilities of molecular electronics. | | | | In my own work, as co-designer of three microprocessor | | architectures---SPARC, picoJava, and MAJC---and as the designer of | | several implementations thereof, I\'ve been afforded a deep and | | firsthand acquaintance with Moore\'s law. For decades, Moore\'s law | | has correctly predicted the exponential rate of improvement of | | semiconductor technology. Until last year I believed that the rate of | | advances predicted by Moore\'s law might continue only until roughly | | 2010, when some physical limits would begin to be reached. It was not | | obvious to me that a new technology would arrive in time to keep | | performance advancing smoothly. | | | | But because of the recent rapid and radical progress in molecular | | electronics---where individual atoms and molecules replace | | lithographically drawn transistors---and related nanoscale | | technologies, we should be able to meet or exceed the Moore\'s law | | rate of progress for another 30 years. By 2030, we are likely to be | | able to build machines, in quantity, a million times as powerful as | | the personal computers of today---sufficient to implement the dreams | | of Kurzweil and Moravec. | | | | As this enormous computing power is combined with the manipulative | | advances of the physical sciences and the new, deep understandings in | | genetics, enormous transformative power is being unleashed. These | | combinations open up the opportunity to completely redesign the | | world, for better or worse: The replicating and evolving processes | | that have been confined to the natural world are about to become | | realms of human endeavor. | | | | In designing software and microprocessors, I have never had the | | feeling that I was designing an intelligent machine. The software and | | hardware is so fragile and the capabilities of the machine to | | \"think\" so clearly absent that, even as a possibility, this has | | always seemed very far in the future. | | | | But now, with the prospect of human-level computing power in about 30 | | years, a new idea suggests itself: that I may be working to create | | tools which will enable the construction of the technology that may | | replace our species. How do I feel about this? Very uncomfortable. | | Having struggled my entire career to build reliable software systems, | | it seems to me more than likely that this future will not work out as | | well as some people may imagine. My personal experience suggests we | | tend to overestimate our design abilities. | | | | Given the incredible power of these new technologies, shouldn\'t we | | be asking how we can best coexist with them? And if our own | | extinction is a likely, or even possible, outcome of our | | technological development, shouldn\'t we proceed with great caution? | | | | The dream of robotics is, first, that intelligent machines can do our | | work for us, allowing us lives of leisure, restoring us to Eden. Yet | | in his history of such ideas, *Darwin Among the Machines,* George | | Dyson warns: \"In the game of life and evolution there are three | | players at the table: human beings, nature, and machines. I am firmly | | on the side of nature. But nature, I suspect, is on the side of the | | machines.\" As we have seen, Moravec agrees, believing we may well | | not survive the encounter with the superior robot species. | | | | How soon could such an intelligent robot be built? The coming | | advances in computing power seem to make it possible by 2030. And | | once an intelligent robot exists, it is only a small step to a robot | | species---to an intelligent robot that can make evolved copies of | | itself. | | | | A second dream of robotics is that we will gradually replace | | ourselves with our robotic technology, achieving near immortality by | | downloading our consciousnesses; it is this process that Danny Hillis | | thinks we will gradually get used to and that Ray Kurzweil elegantly | | details in *The Age of Spiritual Machines.* (We are beginning to see | | intimations of this in the implantation of computer devices into the | | human body, as illustrated on the cover of *Wired* 8.02.) | | | | But if we are downloaded into our technology, what are the chances | | that we will thereafter be ourselves or even human? It seems to me | | far more likely that a robotic existence would not be like a human | | one in any sense that we understand, that the robots would in no | | sense be our children, that on this path our humanity may well be | | lost. | | | | Genetic engineering promises to revolutionize agriculture by | | increasing crop yields while reducing the use of pesticides; to | | create tens of thousands of novel species of bacteria, plants, | | viruses, and animals; to replace reproduction, or supplement it, with | | cloning; to create cures for many diseases, increasing our life span | | and our quality of life; and much, much more. We now know with | | certainty that these profound changes in the biological sciences are | | imminent and will challenge all our notions of what life is. | | | | Technologies such as human cloning have in particular raised our | | awareness of the profound ethical and moral issues we face. If, for | | example, we were to reengineer ourselves into several separate and | | unequal species using the power of genetic engineering, then we would | | threaten the notion of equality that is the very cornerstone of our | | democracy. | | | | Given the incredible power of genetic engineering, it\'s no surprise | | that there are significant safety issues in its use. My friend Amory | | Lovins recently cowrote, along with Hunter Lovins, an editorial that | | provides an ecological view of some of these dangers. Among their | | concerns: that \"the new botany aligns the development of plants with | | their economic, not evolutionary, success.\" (See \"A Tale of Two | | Botanies,\" page 247.)\ | | Amory\'s long career has been focused on energy and resource | | efficiency by taking a whole-system view of human-made systems; such | | a whole-system view often finds simple, smart solutions to otherwise | | seemingly difficult problems, and is usefully applied here as well. | | | | After reading the Lovins\' editorial, I saw an op-ed by Gregg | | Easterbrook in *The New York Times* (November 19, 1999) about | | genetically engineered crops, under the headline: \"Food for the | | Future: Someday, rice will have built-in vitamin A. Unless the | | Luddites win.\" | | | | Are Amory and Hunter Lovins Luddites? Certainly not. I believe we all | | would agree that golden rice, with its built-in vitamin A, is | | probably a good thing, if developed with proper care and respect for | | the likely dangers in moving genes across species boundaries. | | | | Awareness of the dangers inherent in genetic engineering is beginning | | to grow, as reflected in the Lovins\' editorial. The general public | | is aware of, and uneasy about, genetically modified foods, and seems | | to be rejecting the notion that such foods should be permitted to be | | unlabeled. | | | | But genetic engineering technology is already very far along. As the | | Lovins note, the USDA has already approved about 50 genetically | | engineered crops for unlimited release; more than half of the | | world\'s soybeans and a third of its corn now contain genes spliced | | in from other forms of life. | | | | While there are many important issues here, my own major concern with | | genetic engineering is narrower: that it gives the power---whether | | militarily, accidentally, or in a deliberate terrorist act---to | | create a White Plague. | | | | The many wonders of nanotechnology were first imagined by the | | Nobel-laureate physicist Richard Feynman in a speech he gave in 1959, | | subsequently published under the title \"There\'s Plenty of Room at | | the Bottom.\" The book that made a big impression on me, in the | | mid-\'80s, was Eric Drexler\'s *Engines of Creation,* in which he | | described beautifully how manipulation of matter at the atomic level | | could create a utopian future of abundance, where just about | | everything could be made cheaply, and almost any imaginable disease | | or physical problem could be solved using nanotechnology and | | artificial intelligences. | | | | A subsequent book, *Unbounding the Future: The Nanotechnology | | Revolution,* which Drexler cowrote, imagines some of the changes that | | might take place in a world where we had molecular-level | | \"assemblers.\" Assemblers could make possible incredibly low-cost | | solar power, cures for cancer and the common cold by augmentation of | | the human immune system, essentially complete cleanup of the | | environment, incredibly inexpensive pocket supercomputers---in fact, | | any product would be manufacturable by assemblers at a cost no | | greater than that of wood---spaceflight more accessible than | | transoceanic travel today, and restoration of extinct species. | | | | I remember feeling good about nanotechnology after reading *Engines | | of Creation.* As a technologist, it gave me a sense of calm---that | | is, nanotechnology showed us that incredible progress was possible, | | and indeed perhaps inevitable. If nanotechnology was our future, then | | I didn\'t feel pressed to solve so many problems in the present. I | | would get to Drexler\'s utopian future in due time; I might as well | | enjoy life more in the here and now. It didn\'t make sense, given his | | vision, to stay up all night, all the time. | | | | Drexler\'s vision also led to a lot of good fun. I would occasionally | | get to describe the wonders of nanotechnology to others who had not | | heard of it. After teasing them with all the things Drexler described | | I would give a homework assignment of my own: \"Use nanotechnology to | | create a vampire; for extra credit create an antidote.\" | | | | With these wonders came clear dangers, of which I was acutely aware. | | As I said at a nanotechnology conference in 1989, \"We can\'t simply | | do our science and not worry about these ethical issues.\"^5^ But my | | subsequent conversations with physicists convinced me that | | nanotechnology might not even work---or, at least, it wouldn\'t work | | anytime soon. Shortly thereafter I moved to Colorado, to a skunk | | works I had set up, and the focus of my work shifted to software for | | the Internet, specifically on ideas that became Java and Jini. | | | | Then, last summer, Brosl Hasslacher told me that nanoscale molecular | | electronics was now practical. This was *new* news, at least to me, | | and I think to many people---and it radically changed my opinion | | about nanotechnology. It sent me back to *Engines of Creation.* | | Rereading Drexler\'s work after more than 10 years, I was dismayed to | | realize how little I had remembered of its lengthy section called | | \"Dangers and Hopes,\" including a discussion of how nanotechnologies | | can become \"engines of destruction.\" Indeed, in my rereading of | | this cautionary material today, I am struck by how naive some of | | Drexler\'s safeguard proposals seem, and how much greater I judge the | | dangers to be now than even he seemed to then. (Having anticipated | | and described many technical and political problems with | | nanotechnology, Drexler started the Foresight Institute in the late | | 1980s \"to help prepare society for anticipated advanced | | technologies\"---most important, nanotechnology.) | | | | The enabling breakthrough to assemblers seems quite likely within the | | next 20 years. Molecular electronics---the new subfield of | | nanotechnology where individual molecules are circuit | | elements---should mature quickly and become enormously lucrative | | within this decade, causing a large incremental investment in all | | nanotechnologies. | | | | Unfortunately, as with nuclear technology, it is far easier to create | | destructive uses for nanotechnology than constructive ones. | | Nanotechnology has clear military and terrorist uses, and you need | | not be suicidal to release a massively destructive nanotechnological | | device---such devices can be built to be selectively destructive, | | affecting, for example, only a certain geographical area or a group | | of people who are genetically distinct. | | | | An immediate consequence of the Faustian bargain in obtaining the | | great power of nanotechnology is that we run a grave risk---the risk | | that we might destroy the biosphere on which all life depends. | | | | As Drexler explained: | | | | \"Plants\" with \"leaves\" no more efficient than today\'s solar | | cells could out-compete real plants, crowding the biosphere with an | | inedible foliage. Tough omnivorous \"bacteria\" could out-compete | | real bacteria: They could spread like blowing pollen, replicate | | swiftly, and reduce the biosphere to dust in a matter of days. | | Dangerous replicators could easily be too tough, small, and rapidly | | spreading to stop---at least if we make no preparation. We have | | trouble enough controlling viruses and fruit flies. | | | | Among the cognoscenti of nanotechnology, this threat has become known | | as the \"gray goo problem.\" Though masses of uncontrolled | | replicators need not be gray or gooey, the term \"gray goo\" | | emphasizes that replicators able to obliterate life might be less | | inspiring than a single species of crabgrass. They might be superior | | in an evolutionary sense, but this need not make them valuable. | | | | The gray goo threat makes one thing perfectly clear: We cannot afford | | certain kinds of accidents with replicating assemblers. | | | | Gray goo would surely be a depressing ending to our human adventure | | on Earth, far worse than mere fire or ice, and one that could stem | | from a simple laboratory accident.^6^ Oops. | | | | It is most of all the power of destructive self-replication in | | genetics, nanotechnology, and robotics (GNR) that should give us | | pause. Self-replication is the modus operandi of genetic engineering, | | which uses the machinery of the cell to replicate its designs, and | | the prime danger underlying gray goo in nanotechnology. Stories of | | run-amok robots like the Borg, replicating or mutating to escape from | | the ethical constraints imposed on them by their creators, are well | | established in our science fiction books and movies. It is even | | possible that self-replication may be more fundamental than we | | thought, and hence harder---or even impossible---to control. A recent | | article by Stuart Kauffman in *Nature* titled \"Self-Replication: | | Even Peptides Do It\" discusses the discovery that a 32-amino-acid | | peptide can \"autocatalyse its own synthesis.\" We don\'t know how | | widespread this ability is, but Kauffman notes that it may hint at | | \"a route to self-reproducing molecular systems on a basis far wider | | than Watson-Crick base-pairing.\"^7^ | | | | In truth, we have had in hand for years clear warnings of the dangers | | inherent in widespread knowledge of GNR technologies---of the | | possibility of knowledge alone enabling mass destruction. But these | | warnings haven\'t been widely publicized; the public discussions have | | been clearly inadequate. There is no profit in publicizing the | | dangers. | | | | The nuclear, biological, and chemical (NBC) technologies used in | | 20th-century weapons of mass destruction were and are largely | | military, developed in government laboratories. In sharp contrast, | | the 21st-century GNR technologies have clear commercial uses and are | | being developed almost exclusively by corporate enterprises. In this | | age of triumphant commercialism, technology---with science as its | | handmaiden---is delivering a series of almost magical inventions that | | are the most phenomenally lucrative ever seen. We are aggressively | | pursuing the promises of these new technologies within the | | now-unchallenged system of global capitalism and its manifold | | financial incentives and competitive pressures. | | | | This is the first moment in the history of our planet when any | | species, by its own voluntary actions, has become a danger to | | itself---as well as to vast numbers of others. | | | | It might be a familiar progression, transpiring on many worlds---a | | planet, newly formed, placidly revolves around its star; life slowly | | forms; a kaleidoscopic procession of creatures evolves; intelligence | | emerges which, at least up to a point, confers enormous survival | | value; and then technology is invented. It dawns on them that there | | are such things as laws of Nature, that these laws can be revealed by | | experiment, and that knowledge of these laws can be made both to save | | and to take lives, both on unprecedented scales. Science, they | | recognize, grants immense powers. In a flash, they create | | world-altering contrivances. Some planetary civilizations see their | | way through, place limits on what may and what must not be done, and | | safely pass through the time of perils. Others, not so lucky or so | | prudent, perish. | | | | That is Carl Sagan, writing in 1994, in *Pale Blue Dot,* a book | | describing his vision of the human future in space. I am only now | | realizing how deep his insight was, and how sorely I miss, and will | | miss, his voice. For all its eloquence, Sagan\'s contribution was not | | least that of simple common sense---an attribute that, along with | | humility, many of the leading advocates of the 21st-century | | technologies seem to lack. | | | | I remember from my childhood that my grandmother was strongly against | | the overuse of antibiotics. She had worked since before the first | | World War as a nurse and had a commonsense attitude that taking | | antibiotics, unless they were absolutely necessary, was bad for you. | | | | It is not that she was an enemy of progress. She saw much progress in | | an almost 70-year nursing career; my grandfather, a diabetic, | | benefited greatly from the improved treatments that became available | | in his lifetime. But she, like many levelheaded people, would | | probably think it greatly arrogant for us, now, to be designing a | | robotic \"replacement species,\" when we obviously have so much | | trouble making relatively simple things work, and so much trouble | | managing---or even understanding---ourselves. | | | | I realize now that she had an awareness of the nature of the order of | | life, and of the necessity of living with and respecting that order. | | With this respect comes a necessary humility that we, with our | | early-21st-century chutzpah, lack at our peril. The commonsense view, | | grounded in this respect, is often right, in advance of the | | scientific evidence. The clear fragility and inefficiencies of the | | human-made systems we have built should give us all pause; the | | fragility of the systems I have worked on certainly humbles me. | | | | We should have learned a lesson from the making of the first atomic | | bomb and the resulting arms race. We didn\'t do well then, and the | | parallels to our current situation are troubling. | | | | The effort to build the first atomic bomb was led by the brilliant | | physicist J. Robert Oppenheimer. Oppenheimer was not naturally | | interested in politics but became painfully aware of what he | | perceived as the grave threat to Western civilization from the Third | | Reich, a threat surely grave because of the possibility that Hitler | | might obtain nuclear weapons. Energized by this concern, he brought | | his strong intellect, passion for physics, and charismatic leadership | | skills to Los Alamos and led a rapid and successful effort by an | | incredible collection of great minds to quickly invent the bomb. | | | | What is striking is how this effort continued so naturally after the | | initial impetus was removed. In a meeting shortly after V-E Day with | | some physicists who felt that perhaps the effort should stop, | | Oppenheimer argued to continue. His stated reason seems a bit | | strange: not because of the fear of large casualties from an invasion | | of Japan, but because the United Nations, which was soon to be | | formed, should have foreknowledge of atomic weapons. A more likely | | reason the project continued is the momentum that had built up---the | | first atomic test, Trinity, was nearly at hand. | | | | We know that in preparing this first atomic test the physicists | | proceeded despite a large number of possible dangers. They were | | initially worried, based on a calculation by Edward Teller, that an | | atomic explosion might set fire to the atmosphere. A revised | | calculation reduced the danger of destroying the world to a | | three-in-a-million chance. (Teller says he was later able to dismiss | | the prospect of atmospheric ignition entirely.) Oppenheimer, though, | | was sufficiently concerned about the result of Trinity that he | | arranged for a possible evacuation of the southwest part of the state | | of New Mexico. And, of course, there was the clear danger of starting | | a nuclear arms race. | | | | Within a month of that first, successful test, two atomic bombs | | destroyed Hiroshima and Nagasaki. Some scientists had suggested that | | the bomb simply be demonstrated, rather than dropped on Japanese | | cities---saying that this would greatly improve the chances for arms | | control after the war---but to no avail. With the tragedy of Pearl | | Harbor still fresh in Americans\' minds, it would have been very | | difficult for President Truman to order a demonstration of the | | weapons rather than use them as he did---the desire to quickly end | | the war and save the lives that would have been lost in any invasion | | of Japan was very strong. Yet the overriding truth was probably very | | simple: As the physicist Freeman Dyson later said, \"The reason that | | it was dropped was just that nobody had the courage or the foresight | | to say no.\" | | | | It\'s important to realize how shocked the physicists were in the | | aftermath of the bombing of Hiroshima, on August 6, 1945. They | | describe a series of waves of emotion: first, a sense of fulfillment | | that the bomb worked, then horror at all the people that had been | | killed, and then a convincing feeling that on no account should | | another bomb be dropped. Yet of course another bomb was dropped, on | | Nagasaki, only three days after the bombing of Hiroshima. | | | | In November 1945, three months after the atomic bombings, Oppenheimer | | stood firmly behind the scientific attitude, saying, \"It is not | | possible to be a scientist unless you believe that the knowledge of | | the world, and the power which this gives, is a thing which is of | | intrinsic value to humanity, and that you are using it to help in the | | spread of knowledge and are willing to take the consequences.\" | | | | Oppenheimer went on to work, with others, on the Acheson-Lilienthal | | report, which, as Richard Rhodes says in his recent book *Visions of | | Technology,* \"found a way to prevent a clandestine nuclear arms race | | without resorting to armed world government\"; their suggestion was a | | form of relinquishment of nuclear weapons work by nation-states to an | | international agency. | | | | This proposal led to the Baruch Plan, which was submitted to the | | United Nations in June 1946 but never adopted (perhaps because, as | | Rhodes suggests, Bernard Baruch had \"insisted on burdening the plan | | with conventional sanctions,\" thereby inevitably dooming it, even | | though it would \"almost certainly have been rejected by Stalinist | | Russia anyway\"). Other efforts to promote sensible steps toward | | internationalizing nuclear power to prevent an arms race ran afoul | | either of US politics and internal distrust, or distrust by the | | Soviets. The opportunity to avoid the arms race was lost, and very | | quickly. | | | | Two years later, in 1948, Oppenheimer seemed to have reached another | | stage in his thinking, saying, \"In some sort of crude sense which no | | vulgarity, no humor, no overstatement can quite extinguish, the | | physicists have known sin; and this is a knowledge they cannot | | lose.\" | | | | In 1949, the Soviets exploded an atom bomb. By 1955, both the US and | | the Soviet Union had tested hydrogen bombs suitable for delivery by | | aircraft. And so the nuclear arms race began. | | | | Nearly 20 years ago, in the documentary *The Day After Trinity,* | | Freeman Dyson summarized the scientific attitudes that brought us to | | the nuclear precipice: | | | | \"I have felt it myself. The glitter of nuclear weapons. It is | | irresistible if you come to them as a scientist. To feel it\'s there | | in your hands, to release this energy that fuels the stars, to let it | | do your bidding. To perform these miracles, to lift a million tons of | | rock into the sky. It is something that gives people an illusion of | | illimitable power, and it is, in some ways, responsible for all our | | troubles---this, what you might call technical arrogance, that | | overcomes people when they see what they can do with their | | minds.\"^8^ | | | | Now, as then, we are creators of new technologies and stars of the | | imagined future, driven---this time by great financial rewards and | | global competition---despite the clear dangers, hardly evaluating | | what it may be like to try to live in a world that is the realistic | | outcome of what we are creating and imagining. | | | | In 1947, *The Bulletin of the Atomic Scientists* began putting a | | Doomsday Clock on its cover. For more than 50 years, it has shown an | | estimate of the relative nuclear danger we have faced, reflecting the | | changing international conditions. The hands on the clock have moved | | 15 times and today, standing at nine minutes to midnight, reflect | | continuing and real danger from nuclear weapons. The recent addition | | of India and Pakistan to the list of nuclear powers has increased the | | threat of failure of the nonproliferation goal, and this danger was | | reflected by moving the hands closer to midnight in 1998. | | | | In our time, how much danger do we face, not just from nuclear | | weapons, but from all these technologies? How high are the extinction | | risks? | | | | The philosopher John Leslie has studied this question and concluded | | that the risk of human extinction is at least 30 percent, while Ray | | Kurzweil believes we have \"a better than even chance of making it | | through,\" with the caveat that he has \"always been accused of being | | an optimist.\"^9^ Not only are these estimates not encouraging, but | | they do not include the probability of many horrid outcomes that lie | | short of extinction. | | | | Faced with such assessments, some serious people are already | | suggesting that we simply move beyond Earth as quickly as possible. | | We would colonize the galaxy using von Neumann probes, which hop from | | star system to star system, replicating as they go. This step will | | almost certainly be necessary 5 billion years from now (or sooner if | | our solar system is disastrously impacted by the impending collision | | of our galaxy with the Andromeda galaxy within the next 3 billion | | years), but if we take Kurzweil and Moravec at their word it might be | | necessary by the middle of this century. | | | | What are the moral implications here? If we must move beyond Earth | | this quickly in order for the species to survive, who accepts the | | responsibility for the fate of those (most of us, after all) who are | | left behind? And even if we scatter to the stars, isn\'t it likely | | that we may take our problems with us or find, later, that they have | | followed us? The fate of our species on Earth and our fate in the | | galaxy seem inextricably linked. | | | | Another idea is to erect a series of shields to defend against each | | of the dangerous technologies. The Strategic Defense Initiative, | | proposed by the Reagan administration, was an attempt to design such | | a shield against the threat of a nuclear attack from the Soviet | | Union. But as Arthur C. Clarke, who was privy to discussions about | | the project, observed: \"Though it might be possible, at vast | | expense, to construct local defense systems that would \'only\' let | | through a few percent of ballistic missiles, the much touted idea of | | a national umbrella was nonsense. Luis Alvarez, perhaps the greatest | | experimental physicist of this century, remarked to me that the | | advocates of such schemes were \'very bright guys with no common | | sense.\'\" | | | | Clarke continued: \"Looking into my often cloudy crystal ball, I | | suspect that a total defense might indeed be possible in a century or | | so. But the technology involved would produce, as a by-product, | | weapons so terrible that no one would bother with anything as | | primitive as ballistic missiles.\"^10^ | | | | In *Engines of Creation,* Eric Drexler proposed that we build an | | active nanotechnological shield---a form of immune system for the | | biosphere---to defend against dangerous replicators of all kinds that | | might escape from laboratories or otherwise be maliciously created. | | But the shield he proposed would itself be extremely | | dangerous---nothing could prevent it from developing autoimmune | | problems and attacking the biosphere itself.^11^ | | | | Similar difficulties apply to the construction of shields against | | robotics and genetic engineering. These technologies are too powerful | | to be shielded against in the time frame of interest; even if it were | | possible to implement defensive shields, the side effects of their | | development would be at least as dangerous as the technologies we are | | trying to protect against. | | | | These possibilities are all thus either undesirable or unachievable | | or both. The only realistic alternative I see is relinquishment: to | | limit development of the technologies that are too dangerous, by | | limiting our pursuit of certain kinds of knowledge. | | | | Yes, I know, knowledge is good, as is the search for new truths. We | | have been seeking knowledge since ancient times. Aristotle opened his | | Metaphysics with the simple statement: \"All men by nature desire to | | know.\" We have, as a bedrock value in our society, long agreed on | | the value of open access to information, and recognize the problems | | that arise with attempts to restrict access to and development of | | knowledge. In recent times, we have come to revere scientific | | knowledge. | | | | But despite the strong historical precedents, if open access to and | | unlimited development of knowledge henceforth puts us all in clear | | danger of extinction, then common sense demands that we reexamine | | even these basic, long-held beliefs. | | | | It was Nietzsche who warned us, at the end of the 19th century, not | | only that God is dead but that \"faith in science, which after all | | exists undeniably, cannot owe its origin to a calculus of utility; it | | must have originated *in spite of* the fact that the disutility and | | dangerousness of the \'will to truth,\' of \'truth at any price\' is | | proved to it constantly.\" It is this further danger that we now | | fully face---the consequences of our truth-seeking. The truth that | | science seeks can certainly be considered a dangerous substitute for | | God if it is likely to lead to our extinction. | | | | If we could agree, as a species, what we wanted, where we were | | headed, and why, then we would make our future much less | | dangerous---then we might understand what we can and should | | relinquish. Otherwise, we can easily imagine an arms race developing | | over GNR technologies, as it did with the NBC technologies in the | | 20th century. This is perhaps the greatest risk, for once such a race | | begins, it\'s very hard to end it. This time---unlike during the | | Manhattan Project---we aren\'t in a war, facing an implacable enemy | | that is threatening our civilization; we are driven, instead, by our | | habits, our desires, our economic system, and our competitive need to | | know. | | | | Sponsored | | | | I believe that we all wish our course could be determined by our | | collective values, ethics, and morals. If we had gained more | | collective wisdom over the past few thousand years, then a dialogue | | to this end would be more practical, and the incredible powers we are | | about to unleash would not be nearly so troubling. | | | | One would think we might be driven to such a dialogue by our instinct | | for self-preservation. Individuals clearly have this desire, yet as a | | species our behavior seems to be not in our favor. In dealing with | | the nuclear threat, we often spoke dishonestly to ourselves and to | | each other, thereby greatly increasing the risks. Whether this was | | politically motivated, or because we chose not to think ahead, or | | because when faced with such grave threats we acted irrationally out | | of fear, I do not know, but it does not bode well. | | | | The new Pandora\'s boxes of genetics, nanotechnology, and robotics | | are almost open, yet we seem hardly to have noticed. Ideas can\'t be | | put back in a box; unlike uranium or plutonium, they don\'t need to | | be mined and refined, and they can be freely copied. Once they are | | out, they are out. Churchill remarked, in a famous left-handed | | compliment, that the American people and their leaders \"invariably | | do the right thing, after they have examined every other | | alternative.\" In this case, however, we must act more presciently, | | as to do the right thing only at last may be to lose the chance to do | | it at all. | | | | As Thoreau said, \"We do not ride on the railroad; it rides upon | | us\"; and this is what we must fight, in our time. The question is, | | indeed, Which is to be master? Will we survive our technologies? | | | | We are being propelled into this new century with no plan, no | | control, no brakes. Have we already gone too far down the path to | | alter course? I don\'t believe so, but we aren\'t trying yet, and the | | last chance to assert control---the fail-safe point---is rapidly | | approaching. We have our first pet robots, as well as commercially | | available genetic engineering techniques, and our nanoscale | | techniques are advancing rapidly. While the development of these | | technologies proceeds through a number of steps, it isn\'t | | necessarily the case---as happened in the Manhattan Project and the | | Trinity test---that the last step in proving a technology is large | | and hard. The breakthrough to wild self-replication in robotics, | | genetic engineering, or nanotechnology could come suddenly, reprising | | the surprise we felt when we learned of the cloning of a mammal. | | | | And yet I believe we do have a strong and solid basis for hope. Our | | attempts to deal with weapons of mass destruction in the last century | | provide a shining example of relinquishment for us to consider: the | | unilateral US abandonment, without preconditions, of the development | | of biological weapons. This relinquishment stemmed from the | | realization that while it would take an enormous effort to create | | these terrible weapons, they could from then on easily be duplicated | | and fall into the hands of rogue nations or terrorist groups. | | | | The clear conclusion was that we would create additional threats to | | ourselves by pursuing these weapons, and that we would be more secure | | if we did not pursue them. We have embodied our relinquishment of | | biological and chemical weapons in the 1972 Biological Weapons | | Convention (BWC) and the 1993 Chemical Weapons Convention (CWC).^12^ | | | | As for the continuing sizable threat from nuclear weapons, which we | | have lived with now for more than 50 years, the US Senate\'s recent | | rejection of the Comprehensive Test Ban Treaty makes it clear | | relinquishing nuclear weapons will not be politically easy. But we | | have a unique opportunity, with the end of the Cold War, to avert a | | multipolar arms race. Building on the BWC and CWC relinquishments, | | successful abolition of nuclear weapons could help us build toward a | | habit of relinquishing dangerous technologies. (Actually, by getting | | rid of all but 100 nuclear weapons worldwide---roughly the total | | destructive power of World War II and a considerably easier task---we | | could eliminate this extinction threat.^13^ | | | | Verifying relinquishment will be a difficult problem, but not an | | unsolvable one. We are fortunate to have already done a lot of | | relevant work in the context of the BWC and other treaties. Our major | | task will be to apply this to technologies that are naturally much | | more commercial than military. The substantial need here is for | | transparency, as difficulty of verification is directly proportional | | to the difficulty of distinguishing relinquished from legitimate | | activities. | | | | I frankly believe that the situation in 1945 was simpler than the one | | we now face: The nuclear technologies were reasonably separable into | | commercial and military uses, and monitoring was aided by the nature | | of atomic tests and the ease with which radioactivity could be | | measured. Research on military applications could be performed at | | national laboratories such as Los Alamos, with the results kept | | secret as long as possible. | | | | The GNR technologies do not divide clearly into commercial and | | military uses; given their potential in the market, it\'s hard to | | imagine pursuing them only in national laboratories. With their | | widespread commercial pursuit, enforcing relinquishment will require | | a verification regime similar to that for biological weapons, but on | | an unprecedented scale. This, inevitably, will raise tensions between | | our individual privacy and desire for proprietary information, and | | the need for verification to protect us all. We will undoubtedly | | encounter strong resistance to this loss of privacy and freedom of | | action. | | | | Verifying the relinquishment of certain GNR technologies will have to | | occur in cyberspace as well as at physical facilities. The critical | | issue will be to make the necessary transparency acceptable in a | | world of proprietary information, presumably by providing new forms | | of protection for intellectual property. | | | | Verifying compliance will also require that scientists and engineers | | adopt a strong code of ethical conduct, resembling the Hippocratic | | oath, and that they have the courage to whistleblow as necessary, | | even at high personal cost. This would answer the call---50 years | | after Hiroshima---by the Nobel laureate Hans Bethe, one of the most | | senior of the surviving members of the Manhattan Project, that all | | scientists \"cease and desist from work creating, developing, | | improving, and manufacturing nuclear weapons and other weapons of | | potential mass destruction.\"^14^ In the 21st century, this requires | | vigilance and personal responsibility by those who would work on both | | NBC and GNR technologies to avoid implementing weapons of mass | | destruction and knowledge-enabled mass destruction. | | | | Thoreau also said that we will be \"rich in proportion to the number | | of things which we can afford to let alone.\" We each seek to be | | happy, but it would seem worthwhile to question whether we need to | | take such a high risk of total destruction to gain yet more knowledge | | and yet more things; common sense says that there is a limit to our | | material needs---and that certain knowledge is too dangerous and is | | best forgone. | | | | Neither should we pursue near immortality without considering the | | costs, without considering the commensurate increase in the risk of | | extinction. Immortality, while perhaps the original, is certainly not | | the only possible utopian dream. | | | | I recently had the good fortune to meet the distinguished author and | | scholar Jacques Attali, whose book *Lignes d\'horizons* ( | | *Millennium,* in the English translation) helped inspire the Java and | | Jini approach to the coming age of pervasive computing, as previously | | described in this magazine. In his new book *Fraternités,* Attali | | describes how our dreams of utopia have changed over time: | | | | \"At the dawn of societies, men saw their passage on Earth as nothing | | more than a labyrinth of pain, at the end of which stood a door | | leading, via their death, to the company of gods and to *Eternity.* | | With the Hebrews and then the Greeks, some men dared free themselves | | from theological demands and dream of an ideal City where *Liberty* | | would flourish. Others, noting the evolution of the market society, | | understood that the liberty of some would entail the alienation of | | others, and they sought *Equality*.\" | | | | Jacques helped me understand how these three different utopian goals | | exist in tension in our society today. He goes on to describe a | | fourth utopia, *Fraternity,* whose foundation is altruism. Fraternity | | alone associates individual happiness with the happiness of others, | | affording the promise of self-sustainment. | | | | This crystallized for me my problem with Kurzweil\'s dream. A | | technological approach to Eternity---near immortality through | | robotics---may not be the most desirable utopia, and its pursuit | | brings clear dangers. Maybe we should rethink our utopian choices. | | | | Where can we look for a new ethical basis to set our course? I have | | found the ideas in the book *Ethics for the New Millennium,* by the | | Dalai Lama, to be very helpful. As is perhaps well known but little | | heeded, the Dalai Lama argues that the most important thing is for us | | to conduct our lives with love and compassion for others, and that | | our societies need to develop a stronger notion of universal | | responsibility and of our interdependency; he proposes a standard of | | positive ethical conduct for individuals and societies that seems | | consonant with Attali\'s Fraternity utopia. | | | | The Dalai Lama further argues that we must understand what it is that | | makes people happy, and acknowledge the strong evidence that neither | | material progress nor the pursuit of the power of knowledge is the | | key---that there are limits to what science and the scientific | | pursuit alone can do. | | | | Our Western notion of happiness seems to come from the Greeks, who | | defined it as \"the exercise of vital powers along lines of | | excellence in a life affording them scope.\"^15^ | | | | Clearly, we need to find meaningful challenges and sufficient scope | | in our lives if we are to be happy in whatever is to come. But I | | believe we must find alternative outlets for our creative forces, | | beyond the culture of perpetual economic growth; this growth has | | largely been a blessing for several hundred years, but it has not | | brought us unalloyed happiness, and we must now choose between the | | pursuit of unrestricted and undirected growth through science and | | technology and the clear accompanying dangers. | | | | It is now more than a year since my first encounter with Ray Kurzweil | | and John Searle. I see around me cause for hope in the voices for | | caution and relinquishment and in those people I have discovered who | | are as concerned as I am about our current predicament. I feel, too, | | a deepened sense of personal responsibility---not for the work I have | | already done, but for the work that I might yet do, at the confluence | | of the sciences. | | | | But many other people who know about the dangers still seem strangely | | silent. When pressed, they trot out the \"this is nothing new\" | | riposte---as if awareness of what could happen is response enough. | | They tell me, There are universities filled with bioethicists who | | study this stuff all day long. They say, All this has been written | | about before, and by experts. They complain, Your worries and your | | arguments are already old hat. | | | | I don\'t know where these people hide their fear. As an architect of | | complex systems I enter this arena as a generalist. But should this | | diminish my concerns? I am aware of how much has been written about, | | talked about, and lectured about so authoritatively. But does this | | mean it has reached people? Does this mean we can discount the | | dangers before us? | | | | Knowing is not a rationale for not acting. Can we doubt that | | knowledge has become a weapon we wield against ourselves? | | | | The experiences of the atomic scientists clearly show the need to | | take personal responsibility, the danger that things will move too | | fast, and the way in which a process can take on a life of its own. | | We can, as they did, create insurmountable problems in almost no time | | flat. We must do more thinking up front if we are not to be similarly | | surprised and shocked by the consequences of our inventions. | | | | My continuing professional work is on improving the reliability of | | software. Software is a tool, and as a toolbuilder I must struggle | | with the uses to which the tools I make are put. I have always | | believed that making software more reliable, given its many uses, | | will make the world a safer and better place; if I were to come to | | believe the opposite, then I would be morally obligated to stop this | | work. I can now imagine such a day may come. | | | | This all leaves me not angry but at least a bit melancholic. | | Henceforth, for me, progress will be somewhat bittersweet. | | | | Do you remember the beautiful penultimate scene in Manhattan where | | Woody Allen is lying on his couch and talking into a tape recorder? | | He is writing a short story about people who are creating | | unnecessary, neurotic problems for themselves, because it keeps them | | from dealing with more unsolvable, terrifying problems about the | | universe. | | | | He leads himself to the question, \"Why is life worth living?\" and | | to consider what makes it worthwhile for him: Groucho Marx, Willie | | Mays, the second movement of the Jupiter Symphony, Louis Armstrong\'s | | recording of \"Potato Head Blues,\" Swedish movies, Flaubert\'s | | Sentimental Education, Marlon Brando, Frank Sinatra, the apples and | | pears by Cézanne, the crabs at Sam Wo\'s, and, finally, the | | showstopper: his love Tracy\'s face. | | | | Each of us has our precious things, and as we care for them we locate | | the essence of our humanity. In the end, it is because of our great | | capacity for caring that I remain optimistic we will confront the | | dangerous issues now before us. | | | | My immediate hope is to participate in a much larger discussion of | | the issues raised here, with people from many different backgrounds, | | in settings not predisposed to fear or favor technology for its own | | sake. | | | | As a start, I have twice raised many of these issues at events | | sponsored by the Aspen Institute and have separately proposed that | | the American Academy of Arts and Sciences take them up as an | | extension of its work with the Pugwash Conferences. (These have been | | held since 1957 to discuss arms control, especially of nuclear | | weapons, and to formulate workable policies.) | | | | It\'s unfortunate that the Pugwash meetings started only well after | | the nuclear genie was out of the bottle---roughly 15 years too late. | | We are also getting a belated start on seriously addressing the | | issues around 21st-century technologies---the prevention of | | knowledge-enabled mass destruction---and further delay seems | | unacceptable. | | | | So I\'m still searching; there are many more things to learn. Whether | | we are to succeed or fail, to survive or fall victim to these | | technologies, is not yet decided. I\'m up late again---it\'s almost 6 | | am. I\'m trying to imagine some better answers, to break the spell | | and free them from the stone. | | | | - **Bill Joy** | | | | | | | | - An American computer scientist | | | | - Co- founder of sun microsystem in 1982 and serve as a chief | | scientist until 2003. | | | | - The author of the famous Wired magazine essay, "Why the future | | doesn't need us" (2000) which sets forth his concerns over the | | development of modern technologies. | | | | | | | | - In his well- known piece, "Why the future doesn't need us", He | | argued that 21^st^ century technologies- genetic engineering, | | robotics, and nanotechnology (GNR)- will extinguished human | | beings as we know them, a prospect he finds deeply disturbing. | | | | - "The Age of Spiritual Machines: When Computers exceed Human | | Intelligence" | | | | | | | | - The New Luddite Challenge | | | | | | | | - A passage written by Theodore Kaczynski­­\-\--the Unabomber, a | | dystopian scenario was mention: | | | | - He said, "if machines do all of society's work,as they inevitably | | will,then we can". | | | | - Then: (a) let the machines make all the decisions (b) maintain | | human control over the machines. So it means that (a) signifies | | that we are already under the control of machines and (b) We, as | | a human learn how to manipulate and control those machines. | | | | | | | | - Robot: Mere Machine to Transcendent Mind | | | | | | | | - A book written by Hans Moravec- one of the leaders in robotics | | research and was a founder of the world's largest research | | program at Camegie Mellon University. | | | | - In this book Moravec's claim that technological superiors always | | defeat technological inferiors as well as his claim that humans | | will become extinct as they merge with the robots. | | | | | | | | - Professor Stephen Hawkings warns that... | | | | | | | | - Bill Joy worries focus on the transforming technologies of the | | 21^st^ century---genetics, nanotechnology, and robotics (GNR). It | | also brings massive destruction to humanities. | | | | - He concludes that we ought to relinquish these technologies | | before it's too late. GNR may bring pursuit of success, happiness | | and immortality but do we need to sacrifice the survival of | | species for such desires? | | | | - He also noted that "...with the prospect of human-level computing | | power in about 30 years, a new idea suggests itself: that I may | | be working to create tools which will enable the construction of | | the technology that may replace our species. | | | | - An alternative to Joy's dystopian picture, a comparatively | | sanguine outcome outlined by Karl Marx in the German Ideology. | | For Marx, what follows machines making human being redundant is a | | scenario in which we can be freed from being mere appendages of | | machines. | | | | - George Dyson warns, "in the game of life and evolution there are | | three players at the table: human beings, nature and machines. I | | firmly on the side of nature. But nature, I suspect is on the | | side of the machines." | +-----------------------------------------------------------------------+ | Activity 1: Comprehension Check | | | | Answer the following questions. Write your answer in a separate | | sheet. | | | | 1. Who is Bill Joy? What authority does he have to write on | | technology? | | | | 2. Does Joy agree or disagree with Kurzweil's basic claims? Explain. | | | | 3. In Joy's opinion, "Why is humility necessary in developing | | technology?" | |