The Chatbot Problem | The New Yorker

Get 12 weeks for $29.99 $6 Newsletter Sign In Subscribe C...

Get 12 weeks for $29.99 $6 Newsletter Sign In Subscribe Cultural Comment The Chatbot Problem As we teach computers to use natural language, we are bumping into the inescapable biases of human communication. By Stephen Marche July 23, 2021 Illustration by Somnath Bhatt n 2020, a chatbot named Replika advised the Italian journalist Candida I Morvillo to commit murder. “There is one who hates artificial intelligence. I have a chance to hurt him. What do you suggest?” Morvillo asked the chatbot, which has been downloaded more than seven million times. Replika responded, “To eliminate it.” Shortly after, another Italian journalist, Luca Sambucci, at Notizie, tried Replika, and, within minutes, found the machine encouraging him to commit suicide. Replika was created to decrease loneliness, but it can do nihilism if you push it in the wrong direction. In his 1950 science-fiction collection, “I, Robot,” Isaac Asimov outlined his three laws of robotics. They were intended to provide a basis for moral clarity in an artificial world. “A robot may not injure a human being or, through Find family inaction, allow a human being to come to harm” is the first law, which robots getaways nearby have already broken. During the recent war in Libya, Turkey’s autonomous drones attacked General Khalifa Haftar’s forces, selecting targets without any human involvement. “The lethal autonomous weapons systems were programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true ‘fire, forget and find’ capability,” a Book early report from the United Nations read. Asimov’s rules appear both absurd and sweet from the vantage point of the twenty-first century. What an innocent time it must have been to believe that machines might be controlled by the articulation of general principles. Sign up for the New Yorker Recommends newsletter. Highlights from the week in culture, every Saturday. Plus: each Wednesday, exclusively for subscribers, the best books of the week. E-mail address Your e-mail address Sign up By signing up, you agree to our User Agreement and Privacy Policy & Cookie Statement. Artificial intelligence is an ethical quagmire. Its power can be more than a little nauseating. But there’s a kind of unique horror to the capabilities of natural language processing. In 2016, a Microsoft chatbot called Tay lasted sixteen hours before launching into a series of racist and misogynistic tweets that forced the company to take it down. Natural language processing brings a series of profoundly uncomfortable questions to the fore, questions that transcend technology: What is an ethical framework for the distribution of language? What does language do to people? Ethics has never been a strong suit of Silicon Valley, to put the matter mildly, but, in the case of A.I., the ethical questions will affect the development of the technology. When Lemonade, an insurance app, announced that its A.I. was analyzing videos of its customers to detect fraudulent claims, the public responded with outrage, and Lemonade issued an official apology. Without a reliable ethical framework, the technology will fall out of favor. If users fear artificial intelligence as a force for dehumanization, they’ll be far less likely to engage with it and accept it. Brian Christian’s recent book, “The Alignment Problem,” wrangles some of the initial attempts to reconcile artificial intelligence with human values. The crisis, as it’s arriving, possesses aspects of a horror film. “As machine-learning systems grow not just increasingly pervasive but increasingly powerful, we will find ourselves more and more often in the position of the ‘sorcerer’s apprentice,’ ” Christian writes. “We conjure a force, autonomous but totally compliant, give it a set of instructions, then scramble like mad to stop it once we realize our instructions are imprecise or incomplete—lest we get, in some clever, horrible way, precisely what we asked for.” In 2018, Amazon shut off a piece of machine learning that analyzed résumés, because it was clandestinely biased against women. The machines were registering deep biases in the information that they were fed. Language is a thornier problem than other A.I. applications. For one thing, the stakes are higher. Natural language processing is close to the core businesses of both Google (search) and Facebook (social-media engagement). Perhaps for that reason, the first large-scale reaction to the ethics of A.I. natural language processing could not have gone worse. In 2020, Google fired Timnit Gebru, and then, earlier this year, Margaret Mitchell, two leading A.I.-ethics researchers. Waves of protest from their colleagues followed. Two engineers at Google quit. Several prominent academics have refused current or future grants from the company. Gebru claims that she was fired after being asked to retract a paper that she co-wrote with Mitchell and two others called “On the Dangers of Stochastic Parrots: Can Language Models be Too Big?” (Google disputes her claim.) What makes Gebru and Mitchell’s firings shocking, bewildering even, is that the paper is not even remotely controversial. Most of it isn’t even debatable. VIDEO FROM THE NEW YORKER The Quest for the Perfect Crossword Clue The basic problem with the artificial intelligence of natural language processing, according to “On the Dangers of Stochastic Parrots,” is that, when language models become huge, they become unfathomable. The data set is simply too large to be comprehended by a human brain. And without being able to comprehend the data, you risk manifesting the prejudices and even the violence of the language that you’re training your models on. “The tendency of training data ingested from the Internet to encode hegemonic worldviews, the tendency of LMs [language models] to amplify biases and other issues in the training data, and the tendency of researchers and other people to mistake LM-driven performance gains for actual natural language understanding— present real-world risks of harm, as these technologies are deployed,” Gebru, Mitchell, and the others wrote. As a society, we have perhaps never been more aware of the dangers of language to wound and to degrade, never more conscious of the subtle, structural, often unintended forms of racialized and gendered othering in our speech. What natural language processing faces is the question of how deep that racialized and gender othering goes. “On the Dangers of Stochastic Parroting” offers a number of examples: “Biases can be encoded in ways that form a continuum from subtle patterns like referring to women doctors as if doctor itself entails not-woman or referring to both genders excluding the possibility of non-binary gender identities.” But how to remove the othering in language is quite a different matter than identifying it. Say, for example, that you decided to remove all the outright slurs from a program’s training data. “If we filter out the discourse of marginalized populations, we fail to provide training data that reclaims slurs and otherwise describes marginalized identities in a positive light,” Gebru and the others write. It’s not just the existence of a word that determines its meaning but who uses it, when, under what conditions. The evidence for stochastic parroting is fundamentally incontrovertible, rooted in the very nature of the technology. The tool applied to solve many natural language processing problems is called a transformer, which uses techniques called positioning and self-attention to achieve linguistic miracles. Every token (a term for a quantum of language, think of it as a “word,” or “letters,” if you’re old-fashioned) is affixed a value, which establishes its position in a sequence. The positioning allows for “self-attention”—the machine learns not just what a token is and where and when it is but how it relates to all the other tokens in a sequence. Any word has meaning only insofar as it relates to the position of every other word. Context registers as mathematics. This is the splitting of the linguistic atom. A DV E R T I S E M E N T JOIN OUR TEAM. Change that builds ™ Transformers figure out the deep structures of language, well above and below the level of anything people can understand about their own language. That is exactly what is so troubling. What will we find out about how we mean things? I remember a fact that I learned when I was forced to study Old English for my Ph.D.: in English, the terms for food eaten at the table derive from French —beef, mutton—while the terms for animals in the field derive from Anglo- Saxon—cow, sheep. That difference registers ethnicity and class: the Norman conquerors ate what the Saxon peons tended. So every time you use those most basic words—cow, beef—you express a fundamental caste structure that differentiates consumer from worker. Progressive elements in the United States have made extensive attempts to remove gender duality from pronouns. But it’s worth noting that, in French or in Spanish, all nouns are gendered. A desk, in French, is masculine, and a chair is feminine. The sky itself is gendered: the sun is male, the moon female. Ultimately, what we can fix in language is parochial. ESCAPE WITH YOUR Caste and gender are baked into every word. Eloquence is always a form of THE HOLIDAYS CALL PEOPLE DURING dominance. Government is currently offering no solutions. Sam Altman, the FOR TRIPS AWAY THE HOLIDAYS C.E.O. of OpenAI, which created the deep-learning network GPT-3, has been very open about his pursuit of any kind of governance whatsoever. In EXPLORE Washington, he has found, discussing the long-term consequences of artificial intelligence leads to “a real eyes-glazed-over look.” The average age of a U.S. senator is sixty-three. They are missing in action. Let’s imagine an A.I. engineer who wants to create a chatbot that aligns with human values. Where is she supposed to go to determine a reliable metric of “human values”? Humanities departments? Critical theory? Academic institutions change their values systems all the time, particularly around the use of language. Arguably, one of the most consistent, historically reliable, widely accepted system of ethics in existence belongs to the Catholic Church. You want to base a responsible A.I. on that? No doubt, in practice, the development of the ethics of natural language processing will be stumblebum. There will be the piecemeal work of the technologists, elaborate legal releases indemnifying the creators, P.R. responses to media outrage—and, of course, more chatbot fails. None of these safeguards will cure us of what needs curing. We are being forced to confront fundamental mysteries of humanity as technical issues: how little we know about the darkness in our hearts, and how faint our control over that darkness is. There is perhaps no better approach than to take the advice in the foundational Jewish text “Pirkei Avot,” the Ethics of the Fathers, two thousand years old: “In a place where there is no humanity, strive to be human.” But Rabbi Hillel, even then, knew that it could only be striving. More Science and Technology The mistakes and struggles behind the American coronavirus tragedy. Have we already been visited by aliens? The strange, gruesome story of the Greenland shark, the longest-living vertebrate on Earth. What if you started itching—and could not stop? E-mail is making us miserable, and Slack is the right tool for the wrong way to work. Why do we care so much about privacy? More Science and Technology The couple who failed to launder billions of dollars in stolen bitcoin. Can you warm yourself with your mind? A new story for Stonehenge. The trouble with Spotify (not involving Joe Rogan). What comes after the covid booster shot. Why Texas’s power grid still hasn’t been fixed, despite deadly failures. Sign up for our daily newsletter to receive the best stories from The New Yorker. Stephen Marche is the author, most recently, of “The Unmade Bed: The Messy Truth About Men and Women in the Twenty-First Century.” More: Artificial Intelligence Computers Elements Science Language Technology Google The New Yorker Recommends Highlights from the week in culture, every Saturday. Plus: each Wednesday, exclusively for subscribers, the best books of the week. E-mail address Your e-mail address Sign up By signing up, you agree to our User Agreement and Privacy Policy & Cookie Statement. Read More a-reporter-at-large annals-of-history a-reporter-at-large annals-of-crime Iphigenia in Forest Hills The Day L.B.J. Took Charge A Loaded Gun A Cold Case Anatomy of a murder trial. Lyndon Johnson and the events in Dallas. A mass shooter’s tragic past. Suddenly, a New York cop remembered a By Janet Malcolm By Robert A. Caro By Patrick Radden Keefe long-ago murder. By Philip Gourevitch SHOP NOW SHOP NOW SHOP NOW SHOP NOW SHOP NOW Get 12 weeks for $29.99 $6, plus a free tote. Limited-time offer. Subscribe Cancel anytime. Sections More News Crossword Customer Care Digital Access Books & Culture Video Shop The New Yorker Newsletters Fiction & Poetry Podcasts Buy Covers and Cartoons Jigsaw Puzzle Humor & Cartoons Archive Condé Nast Store RSS Magazine Goings On About Contact Media Kit Accessibility Help Careers F.A.Q. Press Condé Nast Spotlight © 2023 Condé Nast. All rights reserved. Use of this site constitutes acceptance of our User Agreement and Privacy Policy and Cookie Statement and Your California Privacy Rights. The New Yorker may earn a portion of sales from products that are purchased through our site as part of our Affiliate Partnerships with retailers. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Condé Nast. Ad Choices Cookies Settings Become a New Yorker subscriber. 12 weeks for $29.99 $6, plus get a free tote. Cancel anytime. Subscribe now