Rationalization Is Rational PDF

Behavioral and Brain Sciences Rationalization is rational cambridge.org/bbs Fiery Cushman Department of Psychology, Harvard University, Ca...

Behavioral and Brain Sciences Rationalization is rational cambridge.org/bbs Fiery Cushman Department of Psychology, Harvard University, Cambridge, MA 02138 [email protected] https://cushmanlab.fas.harvard.edu Target Article Abstract Cite this article: Cushman F. (2020) Rationalization occurs when a person has performed an action and then concocts the beliefs Rationalization is rational. Behavioral and Brain Sciences 43, e28: 1–59. doi:10.1017/ and desires that would have made it rational. Then, people often adjust their own beliefs and S0140525X19001730 desires to match the concocted ones. While many studies demonstrate rationalization, and a few theories describe its underlying cognitive mechanisms, we have little understanding of its Target Article Accepted: 19 May 2019 function. Why is the mind designed to construct post hoc rationalizations of its behavior, and Target Article Manuscript Online: 28 May 2019 Commentaries Accepted: 27 November 2019 then to adopt them? This may accomplish an important task: transferring information between the different kinds of processes and representations that influence our behavior. Human decision making does not rely on a single process; it is influenced by reason, habit, Keywords: instinct, norms, and so on. Several of these influences are not organized according to rational rationalization; cognitive dissonance; self- choice (i.e., computing and maximizing expected value). Rationalization extracts implicit perception; theory of mind; inverse reinforcement learning; habitization; social information – true beliefs and useful desires – from the influence of these non-rational sys- learning; reflective equilibrium; useful fiction; tems on behavior. This is a useful fiction – fiction, because it imputes reason to non-rational representational exchange psychological processes; useful, because it can improve subsequent reasoning. More generally, rationalization belongs to the broader class of representational exchange mechanisms, which transfer information between many different kinds of psychological representations that guide What is Open Peer Commentary? What follows on these pages is known as a our behavior. Representational exchange enables us to represent any information in the man- Treatment, in which a significant and ner best suited to the particular tasks that require it, balancing accuracy, efficiency, and flex- controversial Target Article is published ibility in thought. The theory of representational exchange reveals connections between along with Commentaries (p. 16) and an rationalization and theory of mind, inverse reinforcement learning, thought experiments, Author’s Response (p. 44). See bbsonline. and reflective equilibrium. org for more information. This fight is over. THE MAN standing there. In the silence. Two unconscious cops at his feet. Blood on his pants. What just happened? How did he do this? And there’s THE GUN in his hand. And God, it just feels so natural – checking it – stripping it down – holding it – aiming it – like this is something he’s done a million times before…. This is something he definitely knows how to do. – The Bourne Identity (film) 1. Introduction Jason Bourne is an extraordinary man – a special project of the Central Intelligence Agency. Like a robot, he has been programmed with a vast store of actions that are potentially useful to a clandestine agent. He is fluent in a dozen languages; his gut tells him who to trust and who to fear; he drives like an Italian cabby; he is handy with a gun. But Bourne faces an extraordinary problem. He has lost his memory, identity, goals, and plans – in sum, his ability to make sense of the world and his own place in it. This problem and his solution drive the plot of The Bourne Identity. He must figure out what to believe and what to value by making sense of his peculiar, programmed abilities. The very actions that Bourne performs mindlessly – checking a gun, stripping it, holding it, aiming it – are the clues he uses to rebuild his mind. Jason Bourne learns what to think by seeing what he does. And, in this respect, he is perfectly ordinary. Each of us faces the same problem every day, and each of us grasps for the same solution. We are never fully certain of what to believe and what to value. But by observing the actions we are programmed to perform, we can draw useful inferences – educated guesses about how the world is and what to want from it. Like Bourne’s, ours is a rational project: to reverse-engineer the design principles of our automatic actions. Mercifully, for us, the stakes are usually lower. Perhaps that is why it’s so fun to watch Jason Bourne: His life is ours, just more so. © The Author(s), 2019. Published by Cambridge University Press 1.1. Rationalization Rationalization takes an action that has already been performed and then concocts the beliefs or desires that would have made it rational. It is, therefore, exactly the opposite of rational action1 (Fig. 1). Rational action begins with beliefs and desires and then deduces the optimal https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  2 Cushman: Rationalization is rational action to perform – the one that maximizes desires, conditioned on beliefs. If you believe that a man threatens your life, if you want to live, and if you think he can only be stopped with a bullet, then it is rational to shoot him. Rationalization turns this process on its head: First, you shoot a man, and from this you conclude that he threatened your life. Sensibly or not, people rationalize all the time. Among psy- chologists, it is one of the most exhaustively documented and relentlessly maligned acts in the human repertoire. Classic topics such as cognitive dissonance (Festinger 1962), emotion misattri- bution (Schachter & Singer 1962), appraisal theory (Arnold 1960; Lazarus 1982), self-perception (Bem 1967; Nisbett & Wilson 1977), and confabulation (Gazzaniga 1967) all have ratio- nalization at their heart. More peripherally, it supports confirma- tion bias (Nickerson 1998), system justification (Jost & Banaji 1994), motivated reasoning (Kunda 1990), culpable control (Alicke 2000), hindsight bias (Christensen-Szalanski & Willham 1991), immune neglect (Gilbert et al. 1998), and more. Some cases of rationalization are easy to explain. Perhaps we are merely attempting to explain our own behavior, inferring its obscure causes (Bem 1967). This occurs when, for instance, you notice your furtive glances and flushed cheeks and exclaim to Figure 1. The relationship between (a) rational action and (b) rationalization. yourself: “I’m falling in love!” (Dutton & Aron 1974; Schachter & Singer 1962). Other times, we are hoping to convincingly recast our behavior in a more favorable light (“I ate that last cookie so that nobody else would feel awkward about it!”; Mercier & But these theories mostly fail to explain why the brain would con- Sperber 2011; Tedeschi et al. 1971; von Hippel & Trivers 2011). tain such mechanisms in the first place (but see Mercier & But other cases of rationalization are much harder to explain. Sperber 2011). In other words, why would natural selection In these hard cases people don’t just tell a story, they actually favor a “dissonance reduction motive”? Classic accounts of ratio- make themselves believe it (Brehm 1956; Sharot et al. 2010; nalization are vague – even silent – at this “ultimate” (Tinbergen Vinckier et al. 2019). In one experiment, for instance, participants 1963) or “computational” (Marr 1982) level of analysis. were tricked into believing they had made a subliminal choice To address this challenge, it helps to return to one of most between two vacation destinations, such as Thailand and basic insights of psychological research: Our behavior is influ- Greece. People duped into thinking they chose Greece actually enced by many psychological processes that are (1) unconscious, began to like it more, while people who thought they had chosen (2) non-rational, and yet (3) biological adaptive. For instance, our Thailand showed the opposite preference change (Sharot et al. behavior is influenced by instincts, habits, and conformity to 2010). Similarly, people believe that a lottery ticket is more likely social norms (Fig. 2a). Rationalization, then, may be a mechanism to win as soon as they have bought it (Langer 1975), or that a for extracting valuable information from these adaptive choices horse is more likely to win its race as soon as they have bet on and then allowing it to influence the network of beliefs and it (Knox & Inkster 1968). These cases are hard to see as anything desires that support reasoning (Fig. 2b). but gross errors. You are supposed to choose Thailand because According to this view, rationalization is not merely designed you preferred it, or a bet on horse because of its odds. How, to infer the underlying causes of our behavior for the sake of then, could those choices justify increasing your preference for explanation (Bem 1967). It is not, for instance, designed to dis- Thailand or your belief in the horse’s odds? These cases seem cover our unconscious reasons: hidden beliefs and desires. stubbornly irrational. Why do we drink our own Kool-Aid? Rather, it constructs new beliefs and desires where none had Current theories of rationalization explain how it works, iden- existed, to extract information from the non-rational processes tifying the underlying psychological mechanisms. For instance, that influence our behavior. In other words, just as Jason the theory of cognitive dissonance posits that we revise our pref- Bourne has been programmed by the CIA with a host of useful erences (for Thailand) and beliefs (in a horse’s odds) because we reflexes, we have all been programmed: by natural selection, by are motivated to reduce dissonance between thought and action. habit learning, by social learning, and so forth. Thus, just as Bourne can observe his automatic behaviors and extract useful information, so can we. A simple example illustrates the basic idea. Suppose that an FIERY CUSHMAN is the John L. Loeb Associate Professor of Social infant crawls to high point and then pulls back from the edge Sciences in the Department of Psychology at Harvard University. He by instinct (Gibson & Walk 1960). This action does not reflect is a recipient of the Distinguished Scientific Award for Early Career a belief that heights are dangerous, or the desire to avoid falling; Contribution to Psychology from the American Psychological rather, the infant pulls back from the edge by instinct alone Association, the Stanton Prize from the Society for Philosophy and (Gendler 2008). But, having performed this action, rationalization Psychology, and the Theoretical Innovation Award from the Society seeks to learn from it – first concocting beliefs and desires that for Personality and Social Psychology. His research centers on could have produced it, and then adopting them. For instance, human decision making and moral judgment. infants might conclude that heights are dangerous, or adopt the desire to avoid them. Thus infants do not infer the actual beliefs https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  Cushman: Rationalization is rational 3 Figure 2. Human behavior is influenced by (a) multiple adaptive processes. (b) Rationalization is a method of extracting information from non-rational processes and mak- ing it available to and useful for future reasoning. or desires that guiding their actions; rather, they construct new standard (such as Bayesian inference) in every detail. ones, imputing false mental states to an instinctual action. Nevertheless, the basic structure of rationalization can be under- Nevertheless, their new beliefs and desires are adaptive – stood as an approximation of something rational. precipices are dangerous, and you should avoid them. This is Our first two goals are to review existing theories of rationali- no accident: Our instincts embody the hard-won lessons of natu- zation and to contrast these with the present account. Finally, this ral selection. They are, therefore, a rich source of information for article presents a theory of representational exchange situating your rational mind. rationalization in a broader framework. Representational As adults, of course, we rarely rationalize in situations as sim- exchange describes the flow of information between distinct con- ple as this. For one thing, our behavior is usually the product of trol systems (reasoning, habits, instincts, and norms) to facilitate multiple influences, not a “pure” effect of reflex, habit, reasoning, efficient, adaptive choice. This clarifies the overarching adaptive and so forth. For another thing, our minds are not blank slates rationale that unifies rationalization with many other forms of bereft of prior knowledge and preferences. These facts make representational exchange and highlights its connections to adult rationalization more complex, but no less adaptive. inverse reinforcement learning, habitization, theory of mind, Insofar as non-rational processes exert some influence on our social learning, thought experiments, and the philosophical pur- behavior, and insofar as that influence is adaptive, we can extract suit of reflective equilibrium. useful information by adopting the beliefs and desires that would have made our actions rational. 2. Existing accounts of rationalization Rationalization, then, is “rational” in two senses. First, it is adaptive. This doesn’t guarantee that it always benefits a person The psychological literature on rationalization is large and varied. in every particular case; to the contrary, many psychologists Currently, three basic approaches dominate: (1) cognitive disso- have made their careers by brilliantly illustrating the ways in nance and consonance, (2) self-perception, and (3) persuasion which it can fail. Like any process that is generally adaptive, it and impression management. These theories each differ from will be occasionally be maladaptive. But on average, over time, the current proposal, but do not necessarily compete with it. it pays. For one thing, different kinds of rationalization may occur in dif- Second, at a more specific level, rationalization approximates ferent contexts. More importantly, existing theories mostly inverse planning models of mental state inference (Baker et al. describe the mechanisms of rationalization, whereas the present 2009; Ng & Russell 2000). Cast as a form of Bayesian inference, theory addresses its function. These levels of explanation are inverse planning is sometimes regarded as a rational cognitive often complementary (Marr 1982; Tinbergen 1963). The goal, process. Of course, the particular mechanisms we use to rational- therefore, is not adjudicate between theories, marshaling data ize our behavior are unlikely to conform to a normative, rational for some and against others. Rather, it is to clearly present each https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  4 Cushman: Rationalization is rational theory, and then to consider their points of convergence and eat by saying, “We are motivated by hunger.” Although true divergence. This discussion focuses squarely on the rationalization and important, such a theory is incomplete: It does not explain of action, touching only superficially on other kinds, such as ratio- why the hunger motive evolved (i.e., because food provides the nalizing one’s beliefs or attitudes. raw material for metabolism, and that therefore organisms that possess the hunger motive tend to outcompete those that lack it). Festinger was merely the first in a long line of theorists to 2.1. Dissonance and consonance explain rationalization while eliding its adaptive function Rationalization is usually explained by positing a desire for conso- (Harmon-Jones & Mills 1999): Aronson (1968) proposed that dis- nance between thought and action. The simplest model of this sonance arises most powerfully when actions are incongruent kind, balance theory (Heider 1958/2013), posits that people with a person’s self-concept; Steele (1988; Steele et al. 1993) pro- want to achieve harmony among their attitudes. Thus, if you posed that dissonance is aversive because people feel that appar- like your spouse and your spouse likes guacamole, you will tend ently irrational actions threaten their self-image or self-worth; to acquire a taste for guacamole (or, in theory, a distaste for Beggan (1992) extended this concept to the objects people pos- your spouse). Unfortunately, this model’s generality is also its sess, even when not freely chosen. There are still other possibili- Achilles’ heel: It is easy to come up with compelling counterexam- ties: Perhaps dissonance minimizes post-decisional regret.4 Each ples and hard for the theory to explain them away. As Festinger of these proposals elaborates on Festinger’s mechanistic account, (1999) quipped, “I like chicken, chickens like chicken food, and but they do not offer an ultimate, adaptive explanation for I don’t like chicken food.” rationalization. Festinger’s (1962) and Festinger et al.’s (1956) own theory of Indeed, among the classic approaches, the descendants of rationalization, cognitive dissonance, was therefore more specific. Heider’s balance theory come closest to an adaptive rationale. It posits that only certain sets of beliefs, desires, and actions can Balance among beliefs and attitudes can be formalized as a form occupy states of consonance or dissonance. Although Festinger of logical consistency or constraint satisfaction (Read & did not describe the principle of rational action as such, he Marcus-Newhall 1993; Shultz & Lepper 1999; Thagard 1989). If a grasped it intuitively:2 People tend to act in a way that maximizes person represents that Socrates is a man, and that all men are mortal, their desires, consistent with their beliefs. Sets of beliefs, desires, but that Socrates is immortal, something must give. A well-designed and actions that fit this specific principle are consonant; sets system will repair such inconsistencies in a manner that makes its that do not are dissonant.3 Crucially, Festinger also proposed representations more accurate (Ackley et al. 1985; see also a review that the state of dissonance is psychologically aversive, motivating by Gawronski et al. 2018).5 This idea of network repair or cognitive people to achieve consonance. If you have already acted (e.g., consistency can be fruitfully applied to networks of interrelated shooting a man), of course, it is too late to adjust your action. beliefs or desires (Cushman & Paul, in press). Instead, consonance must be achieved by adjusting your beliefs But how could it be applied to classic cases of rationalization, in or desires (e.g., deciding he must have been a threat). This is which a person revises their beliefs and desires to match their own the essence of rationalization. past action? Suppose that you desire cookies and believe them to Cognitive dissonance is the best-known psychological theory be in the kitchen, but go to the living room. Clearly, one adaptive of rationalization; indeed, it is among the best-known psycholog- response to this imbalance is to correct your action: To go to the ical theories of anything. Its two main pillars enjoy strong empir- kitchen. This applies the principle of rational action. But now con- ical support. First, people change their preferences and beliefs to sider the outcome of rationalization: You either decide you didn’t match actions they have already performed. Specifically, they want cookies in the first place or convince yourself that they are adopt the beliefs and desires that would have made their past actually in the living room. Although this achieves coherence of action rational. A classic method that produces such effects, the a kind, it certainly does not improve your desires or beliefs. free choice paradigm (Brehm 1956), is still widely used (e.g., Rather, it takes take a clear error of reasoning and then multiplies Sharot et al. 2009; 2010; Vinckier et al. 2019). People are given it, infecting thought with a pathology of choice. a choice between two things, such as a toaster and a radio, that Rationalization would indeed be counterproductive if our they value roughly equally. The act of choosing one of these actions were only produced by sound reasons or outright errors. things causes them to value that thing more and the other In this case, dissonance would only arise in cases of error – thing less. This occurs from an early age and in non-human pri- after all, sound reasoning cannot produce actions that violate mates (Egan et al. 2007), even when people cannot remember principles of rationality. And, if rationalization only applied to what they chose (Lieberman et al. 2001), and even when the errors, then it would propagate those errors from action back to experimenter tricks them into believing they chose something our beliefs and desires – a counterproductive result. they didn’t (Sharot et al. 2010). People’s justifications for such Crucially, however, our behavior is influenced by sources other choices can be detailed and elaborate (Johansson et al. 2005). than reason and error. Whether innately, through habit learning, Second, experiments confirm that dissonance is psychologically or through cultural learning, our behavior is influenced by pro- aversive: People say so (Elliot & Devine 1994), and it is also cesses that are adaptive and yet non-rational. Because these pro- revealed by convert measures such as affect misattribution cesses share the same ultimate purpose of reasoning – fitness (Losch & Cacioppo 1990; Zanna & Cooper 1974), psychophysiol- maximization – the beliefs and desires that support reasoning ogy (Losch & Cacioppo 1990; Harmon-Jones et al. 1996), and can be improved by learning from the behavioral influence of functional neuroimaging (van Veen et al. 2009). In sum, the the- non-rational sources. ory is well known for a good reason. Yet, while the theory of cognitive dissonance describes the psy- 2.2. Self-perception chological mechanisms involved in rationalization, it does not offer an ultimate explanation – an answer to the question, Bem’s (1967) theory of self-perception provides quite a different “Why did it evolve?” It is akin to a theory that explains why we explanation for rationalization. It denies the two mechanistic https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  Cushman: Rationalization is rational 5 pillars of cognitive dissonance: First, that dissonance is psycholog- processes are ultimately trying to maximize the same goal: biolog- ically aversive and, second, that underlying desires or beliefs actu- ical fitness. In other words, rationalization is a fiction, but a decid- ally change. Rather, it posits that people only change their edly useful one. Mixed right, it can be nourishing to drink your perceived beliefs and desires. own Kool-Aid. Bem’s original statement of this theory was heavily influenced by behaviorism in two ways. First, he assumed that we have no 2.3. Responsibility avoidance and impression management direct introspective access to the mechanisms that produce our behaviors. Second, he claimed that we receive strong reinforce- Finally, some theories posit that rationalization is designed not so ment from others when we can explain our behavior in terms much to inform others as to persuade them, casting your behavior of mental states, and consequently, we often construct such men- (or other information) in favorable and possibly deceptive light talistic explanations. Desiring mentalistic explanations for our (Tedeschi et al. 1971; von Hippel & Trivers 2011). When risking behavior, but lacking introspective access, we resort to our best blame, for instance, we may profess benign motives or faultless guess: post hoc rationalization. naiveté, even at the expense of the truth. Although intended as an alternative to cognitive dissonance, Mercier and Sperber (2011) go so far as to claim that reasoning self-perception theory stands in its own right as a powerful state- itself is principally adapted to the problem of changing others’ ment: People are unaware of the causes of their behavior and often minds, and therefore they interpret rationalization as an adaptive attempt to infer these causes by observing their actions. These solution to the problem of winning arguments (see also Haidt basic claims are supported by a wealth of experimental research 2001; Tetlock 2002). Indeed, on their view, reasoning itself is (Devine 1989; Gazzaniga 1967; Greenwald & Banaji 1995; Haidt mostly an instance of rationalization. Its goal is to present infor- 2001; Miller & Buckhout 1962/1973; Neisser 1967/2014; Nisbett mation to another person in a manner that compels them, by & Wilson 1977; Wilson 2004). Bem’s (1967) statement of this the- logic or intuition, to accept your conclusion (see also Haidt 2001). ory may have been influenced by seminal work on emotion misat- At first blush, such theories seem to explain only why we tribution (Schachter & Singer 1962) and confabulation in split express rationalization to others, but not why we adjust our own brain patients (Gazzaniga 1967). Nisbett and Wilson (1977) later beliefs or desires (Tedeschi et al. 1971). Yet, it is also plausible condensed these varied insights into three core claims: People that “true believers” are better deceivers. In other words, the are often unaware of the causes of their behavior; self-reports of best way to convince others that you shot an (innocent) man the causes of behavior are generated by folk-causal theories; and, for good reason might be to first convince yourself of his guilt therefore, when they correctly report the cause of their behavior, (Trivers 2000; von Hippel & Trivers 2011). it is usually the result of inference, not introspection. This family of theories likely explains a part of the function of In sum, there is undeniably something right about the theory rationalization. If the present account also explains a part, then it of self-perception. Yet, when aimed at the topic of rationalization, is a complementary but largely independent explanation. it misses two key empirical marks. First, dissonance induces an aversive psychological state that motivates rationalization. 3. Rationalization as construction Second – and of greatest importance – rationalization changes people’s actual beliefs and desires, not just their self-perception. More than a century of research shows that our behavior is influ- For instance, in the free choice paradigm, objects are not just enced by multiple processes (Dolan & Dayan 2013; Kahneman reported to have higher value after being chosen (or lower after 2011b; Thorndike 1898). One influence is rational planning: con- rejection), but they are also actually chosen more often in the sidering the likely outcomes of our behavior according to our future (or less often, after rejected). beliefs, and then choosing the behavior most likely to maximize Like Festinger, Bem did not address what ultimate adaptive our desires. Other influences on our behavior, however, are not purpose might be fulfilled by cognitive dissonance. Instead, he organized according to the principle of rational action. focused on its proximate psychological motivation: The rein- A potential function of rationalization, then, is to construct forcement of social partners who demand mentalistic explana- beliefs and desires that are consistent with the adaptive behaviors tions of our behaviors. Neither did he squarely address the generated by non-rational processes, and then to adopt them. In issue of whether self-perception is usually accurate or inaccurate. other words, like Jason Bourne, rationalization generates new, In the grip of behaviorism, Bem may have regarded this as beside useful insights by observing the actions we perform thoughtlessly. the point – he was likely skeptical that our behavior relies on Later we will view this through a Bayesian lens, as an inversion of structured mental representations at all. Rather, the essence of a generative model of rational action (Baker et al. 2009) and thus a Bem’s claim was simply that we attribute mental states to our- variety of “inverse reinforcement learning” (Ng & Russell 2000). selves by the same processes that we attribute mental states to To explore the logic of rationalization in more detail, it helps others. to focus on three potentially non-rational influences on behav- The present account builds on important insights of self- ior: instincts (innate influences on behavior), conformity to perception theories (Bem’s, and those that followed it), but with social norms (socially learned influences on behavior), and hab- two crucial modifications. First, it posits that the function of self its (reinforced behaviors). Although highly simplified, this taxon- “perception” is not merely to satisfy our own curiosity or that of omy reveals some important insights about the general structure of our peers. It also constructs new beliefs and desires based on rationalization, as well as its specific application in different set- information implicit in other adaptive control mechanisms. tings. All three discussions depend on the common assumption Thus, “perception” is a misnomer: Rationalization is designed that non-rational influences on our behavior are nevertheless not to accurately infer unconscious mental states, but to construct adaptive. This is a natural assumption, given that our instincts, new ones; it is not a discovery, but a fiction. Second, we can, habits, and norms are all processes shaped by adaptive forces: bio- should, and do actually adjust our beliefs and desires to match logical evolution, reinforcement learning, and cultural evolution, this fiction. This is adaptive because reasoning and non-rational respectively. https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  6 Cushman: Rationalization is rational 3.1. Rational action as planning The construction of new beliefs and desires should presumably be structured as a form of rational inference. In Bayesian terms, Before considering how instincts, norms, and habits can improve the posterior beliefs about snakes (“snakes bite” vs. “snakes reasoning, we must have a clearer image of how reasoning itself shoot crossbows”) should be sensitive not just to the likelihood works. Reasoning, sometimes called planning, chooses actions of an action (recoiling) given a percept (snake) and candidate by expected value maximization (Fig. 1a). A simplified model of beliefs (e.g., “snakes bite” vs. “snakes shoot crossbows”), but planning has three parts. First, there is a mechanism for learning also the prior probability of the candidate beliefs, including a causal model of the world, one’s beliefs. This model allows you their compatibility with other beliefs (e.g., animals can’t use cross- to predict what is likely to occur in different situations, depending bows; many long, straight things are perfectly safe). in part on your own actions. Second, there is a mechanism that Indeed, these pieces of information may be integrated in a ratio- assigns intrinsic value to certain outcomes, one’s desires (also nal manner, according to Bayes’ rule. This form of inference has sometimes described as reinforcement or reward). Third, there been well characterized in models of inverse planning (Baker is a mechanism that chooses actions by maximizing the satisfac- et al. 2009; Ng & Russell 2000). This brings into focus an important tion of your desires, given your beliefs. Our next goal is to under- dimension of the claim that rationalization is rational – it is not just stand how a system designed this way could extract useful biologically adaptive, but it may also approximate a well-understood information from instincts, norms, and habits. form of rational inference. Importantly, however, the approximation of a rational inference (at Marr’s computational level) may be quite 3.2. Instincts cognitively simple (at Marr’s algorithmic level). These relationships, between rational inference and the actual mechanisms of rationali- Instincts are innate influences on behavior, designed by natural zation, are discussed more fully in section 4. selection, that bias certain actions to be performed in the presence of certain stimuli.6 For instance, humans instinctively drink when they are thirsty, flee from threats or fight them, reject likely path- 3.4. Norms ogens, fall in love with other humans, and so on. Many of these Human psychology is influenced not just by the biological inher- examples involve very abstract actions (flee) or stimuli (threat). itance of natural selection but also by a vast cultural inheritance. Instincts need not be low-level or concrete, or grounded in a And just as biological natural selection ensures that instincts will single, well-defined neural mechanism. Rather, instinct often typically be adaptive, cultural selection ensures that norms will describes a very abstract kind of innate mental organization. Its typically be adaptive (Boyd et al. 2011), although maladaptations key property is just that the relationship between the stimuli may arise in each case. Our cultural inheritance takes many forms: and the actions is innate and direct. For instance, the perception concepts (π), artifacts (knives), beliefs (the earth is round), desires of a threat may directly bias action toward flight. This is what (money), norms (drive right, pass left), and much more. The spe- makes instincts different from planning, which would instead cific form of norms is often transmitted by social conformity. require a computation like “fleeing avoids threats, threats might These operate analogously to instincts: just as instincts are innate harm me, and I don’t like being harmed.” biases on action, norm conformity may be defined as a set of Because instincts are shaped by natural selection, they tend to socially learned biases on action. increase our biological fitness. Similarly, rational planning is As with instincts, norms may be very abstract. For instance, designed to increase biological fitness. This is an important part there are cultural norms of cooperation and fairness that general- of why rationalization makes sense: It extracts information from ize over many diverse features of specific cases. Norms may also one adaptive system (instinct) and makes it available to another be redundant with other kinds of cultural influence. For instance, (rational planning). If a person’s instinct is incongruent with somebody might comply with the Jewish laws of kashrut because her beliefs and desires, adjusting those beliefs and desires to (1) they wish to get along with their religious peers, or (2) they match her action may ultimately improve them. believe that God asks this of them, or (3) it just feels like the For instance, suppose that a person instinctively recoils from right thing to do. These are, in fact, independent and redundant snakes. This instinct is adaptive because many snakes are venom- elements of cultural learning. According to our restrictive defini- ous, but she happens to be unaware of this. Rationalizing her tion of norms, only the third influence is sufficiently direct to instinct (i.e., attempting to explain her act of recoiling in terms count as a norm. The first two – a desire to get along and a belief of beliefs and desires), she might adopt the belief that snakes are about God – instead influence her behavior indirectly, by dangerous. Similarly, she could rationalize her behavior by adopt- reasoning. ing a general desire to be far from snakes, and this is a useful desire. Given this homology between instincts and norms, the very In either case, the outcomes of her future reasoning are improved. same logic that makes instinct a useful target of rationalization therefore also applies to norms. When a person performs a behav- ior due to cultural influences, she may often be able to extract use- 3.3. Rationalization as a form of rational inference ful beliefs and desires by rationalizing her action. Even if rationalization could possibly generate true beliefs and use- Among the indigenous people of Fiji, for instance, it is taboo ful desires, what guarantees that it would do so typically? The spe- to eat certain kinds of seafood when pregnant or nursing cifics of the snake case are suspiciously convenient; this naïf might (Henrich & Henrich 2010). Most of the taboo seafoods are have concluded instead that snakes breathe fire, shoot crossbows, or toxic and pose special risks to fetuses and infants, but the people dredge up hurtful memories of adolescence. Such beliefs would of Fiji do not have precise knowledge of this – indeed, the taboo explain one’s instinctive recoiling, but they are false. Or she extends to several closely related seafoods that are actually harm- might have adopted the desire to avoid all animals or all things less. Rather, most mothers avoid eating these foods simply that are long and straight. What processes could ensure that the because of the norm. Rationalization, however, might lead a rationalized beliefs and desires are, in fact, useful ones? mother to the correct belief that these fish are dangerous. Then, https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  Cushman: Rationalization is rational 7 Figure 3. Two parallel pathways of social learning: one characterized by theory of mind and the other characterized by rationalization. by reasoning, she may generalize other useful conclusions: don’t alternative function of rationalization: to construct representations even touch these fish, don’t let your baby eat them, don’t feed that are implied by behavior (one’s own, or another person’s). them to sick people, and so forth. This process of construction need not result in a perfectly accurate representation of the causes of behavior in order to be useful (a point made by Dennett in introducing the intentional 3.5. A “useful fiction”: How theory of mind supports social stance7). To the contrary, rationalization can extract useful beliefs learning and desires from the influence of non-rational systems – systems whose influence on behavior had nothing to do with those beliefs Because norms are commonly transmitted via observation and and desires. Rationalization is, in this sense, a useful fiction. It takes imitation (Boyd et al. 2011; Cialdini & Goldstein 2004), the ben- the form of inference, but with a very different function. efits of rationalization one’s own behavior can also be obtained by This same function can also apply to theory of mind. Theory rationalizing others’ behaviors. Put more simply, there is a deep of mind may often involve useful fictions, in which we ascribe homology between rationalization and theory of mind (Fig. 3; inaccurate causes to others’ behavior – goal-directed plans, see also Bem 1967). This motivates a brief but important detour based on beliefs and desires – even when those behaviors were to consider the relationship between theory of mind, self- produced by non-rational processes.8 Although inaccurate, such perception, and rationalization. ascriptions could still extract useful information for us: true beliefs Consider again the seafood taboos of Fiji. A mother might and adaptive desires. conclude that taboo seafood (suppose it is shellfish) is dangerous This perspective has at least one attractive feature: Useful or not, to infants by either of two paths: one via rationalization and theory of mind seems to involve a great deal of fiction. Despite another via theory of mind. Following the first path, she first com- widespread consensus that human behavior is not exclusively ratio- plies with the norm itself, avoiding the shellfish simply because nal, nearly all studies of mental state inference posit a folk theory of others do. Next, observing her own behavior, she rationalizes rational action. Despite more than 40 years of study, there is virtu- that shellfish must be dangerous. Following the second pathway, ally no research on folk theories of instinct, habit, reflex, and the she instead first attempts to understand the behavior of her social like – in other words, a theory-of-the-rest-of-our-minds. Moreover, partners. This act of mental state inference, or theory of mind, what little research exists suggests that people interpret others’ really just amounts to rationalization of others’ behavior. She con- actions as the product of goal-directed reasoning far more than it cludes that her social partners must believe that shellfish are dan- actually is the cause (Gershman et al. 2016). Similarly, experimental gerous. Then, assuming that they know something she doesn’t, demonstrations of automatic behavior are often surprising to lay she adopts this belief herself. audiences, while experimental demonstrations of rational behavior Each of these paths involves a crucial step in which a belief is are not. On the useful fiction model, this is because theory of mind extracted from an action. In the first path it is the observer’s own is not only designed to infer the true causes of a person’s behavior, action, so we call it rationalization; in the second path it is another but also to extract useful beliefs and desires from their behavior person’s action, so we call it theory of mind. (This connection, of even when it was caused by non-rational processes. It is, therefore, course, originates with self-perception theory; Bem 1967). biased to perceive all behavior as rational, even though much This clarifies two important but distinct functions of theory of behavior is not. mind. Many past treatments assume that theory of mind is designed to infer the true causes of another person’s behavior (Baron-Cohen 1995; Dennett 1987; Gopnik et al. 1997). Likewise, 3.6. Habits self-perception theory might be construed as an attempt to accu- rately infer the true cause of one’s own behavior (although Bem Habits are a third major non-rational influence on behavior. himself was agnostic on this point). But we have emphasized an Habits are learned stimulus-response mappings, often reinforced https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  8 Cushman: Rationalization is rational by reward and punishment (reviewed in Dolan & Dayan 2013). who habitually eats cake. Rationalizing this behavior, he con- For instance, a person might habitually flip the lights on when cludes, “I like cake.” Although true, isn’t this information redun- they walk into a room because it is typically useful (i.e., reward- dant? He ought to have discovered that he liked cake back when ing). Each time that the behavior is performed and rewarded, he took his first bite. habit learning strengthens the mapping from stimulus to The answer to this challenge depends on a key insight regard- response. As a result, executing habitual action requires little cog- ing the nature of value-guided learning and decision making. nitive effort. A person does not have to consider desires (“I need Often our behavior is organized sequentially, with early instru- light”) and beliefs (“switches cause light”) to derive the value of mental actions chosen because they eventually bring us to intrin- performing an action; rather, the behavior is habitized based on sically rewarding states of affairs (Bellman 1954). To plow, sow, its value in the past. But, for the same reason, habitual control and harvest are instrumentally valuable actions, for instance, can be inflexible. You might habitually switch the lights on because they ultimately bring a rewarding feast. A major chal- even though you are walking into the room of a sleeping baby lenge, then, is to discover or estimate the instrumental value of and want it to be dark. Rational planning, in contrast, can flexibly various actions. This challenge is especially obvious in games adjust to new or unusual circumstances. like chess: We are attempting to learn the instrumental value of Despite these differences, there are two key similarities moves (or sequences of moves), which is defined by their proba- between habit and reasoning: Both involve learning from direct bility of ultimately attaining checkmate. experience, and both are sensitive to the same rewards. These sim- Habit and reason estimate value in different ways. Habit learn- ilarities make it challenging to explain how the rationalization of ing involves a backward-looking assignment of value: We wait habitual action could provide new information to a system of rea- until checkmate is achieved and then reinforce the sequence of soning. The challenge has two parts: how to improve beliefs and moves that brought us there (Bayer & Glimcher 2005; Glimcher how to improve desires. 2011; Morris et al. 2006; Roesch et al. 2007). In contrast, reason- ing involves a forward-looking assignment of value: We mentally 3.6.1. How to improve beliefs simulate hypothetical future sequences of moves, attempting to Any experience that trains a habit also ought to inform our divine whether they are likely to achieve our goal (Dolan & beliefs, and thus it is not clear why our habits would imply useful Dayan 2013; Sutton & Barto 1998). beliefs that we would not already represent explicitly. For instance, A further benefit of rationalizing habits that depends upon the the experiences that formed your habit of turning on the lights hierarchical nature of human planning (Badre & Nee 2017; ought to have also taught you that flipping the switch makes Botvinick 2008; Norman & Shallice 1986). For instance, if our the light turn on – the very belief that you need to flip on the goal is to make coffee, we plan by calling to mind a series of sub- lights by reasoning. In this case, there is no extra information goals (grind beans, get filter, heat water, etc.), which may them- for your system of goal-directed reasoning to extract. selves contain subgoals (turn on the faucet, turn on the kettle, This first challenge has a few simple replies. First, a person etc.). The essential properties of these subgoals are that they are may simply have forgotten certain facts, and yet nevertheless instrumentally valuable given the superordinate goal, and also have retained an adaptive habit. We have all had the experience that they support generalization across diverse circumstances. of being asked for our opinion on something – a restaurant, a But discovering this form of instrumental value does not come book, a colleague – and being able to recall the valence of our feel- for free – indeed, a major challenge for current theories of hierar- ings (“I know I liked it”; “something about him gave me the chical planning is to explain how we discover the appropriate ways heebie-jeebies”) without being able to recall what was eaten, to carve a task into subgoals (Botvinick 2008; Botvinick & read, or spoken. Even after every detail of an experience evapo- Weinstein 2014; Sutton et al. 1999). rates from memory, the residue of our attitudes may remain. A crucial function of rationalizing habits, then, may be to According to contemporary theories, this residue – the cached translate the instrumental value representations of the habitual values of objects, events, or actions – is the basis of habits system into goal (Keramati et al. 2016) or subgoal (Cushman & (Dolan & Dayan 2013). By rationalizing habits, we can recon- Morris 2015) representations useful to the goal-directed system. struct the details that most likely explain them. For instance, if a tennis player habitually rushes to volley at the Second, and relatedly, a person may have failed to ever formu- net after serving, this likely reflects the instrumental value of late the relevant belief (e.g., because they were not paying atten- serve-and-volley for winning a point. When rationalizing this behav- tion) and yet still have acquired an adaptive habit. Just as it is ior, she may say, “my goal was to gain an advantage over my oppo- familiar to have forgotten why we loved a movie or distrusted a nent while he was on his heels, in order to quickly win the point.” If person, it is equally familiar to never have been quite sure in she internalizes this subgoal, what has she gained? Not a change to the first place. This is possible because habits and beliefs are the value of winning the point (which both systems represented) or learned by distinct and dissociable processes (Foerde & a change to the cached value of serve-and-volley (which the habitual Shohamy 2011; Foerde et al. 2013; Knowlton et al. 1996). system represented), but a novel subgoal representation (“try to serve-and-volley!”) that can improve future planning. 3.6.2. How to improve desires In sum, because values are hard to accurately estimate, and The second and more profound challenge is to explain how because habit and reason estimate value in different ways, the adaptive desires could also be extracted by rationalization. This desires implicated by habitual action may improve our ability to challenge is harder because both systems – habit and reasoning – maximize reward by reasoning. are assumed to begin with the very same set of basic desires: “rewards” (Sutton & Barto 1998) or “primary reinforcers” 3.7. Hybrid control (Kelleher & Gollub 1962), as they are often called. What informa- tion, then, could the habitual system encode that would not Although it is convenient to act as if certain actions are wholly already be encoded by the reasoning system? Consider a person under habitual control, others wholly instinctual, and so on, https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  Cushman: Rationalization is rational 9 this is a caricature. Even the simplest targets of rationalization stud- cognitive operations that facilitate the flow of information ied in the laboratory – the choice of a toaster over a radio, for among distinct systems of behavioral control. instance – are not the product of pure instinct, habit, or norm com- pliance. Rather, they involve at least some degree of conscious, 4. A theory of representational exchange deliberative planning (“let’s see, what could I do with a new toaster?”). More generally, it is disputed whether systems of Rationalization extracts information from non-rational systems habit, instinct, or norm adherence could be cleanly severed from and makes it available to reasoning. It is apparent how this reasoning at all (Dayan 2012; Graybiel 2008; Kool et al. 2018). could improve reasoning, but why would it improve the overall Yet, while reasoning often contributes to choice, it rarely oper- welfare of the organism? In the end, what matters to an organism ates alone. Rather, most behavior is the result of some form of is not to have true beliefs and useful desires, but to perform the approximate planning – an elaborate background of automatic right actions. Insofar as our actions are already appropriately and non-rational processes that construct a restricted and tracta- guided by non-rational forces (habits, instincts, and norms), ble decision space in which limited rational planning can effec- what extra advantage do we gain by improving beliefs and desires? tively guide behavior (Cushman & Morris 2015; Dayan 2012; Properly addressing this question leads to a theoretical frame- Gigerenzer & Selten 2002; Huys et al. 2015; Keramati et al. work that encompasses far more than rationalization. 2016; Kool et al. 2018). Instinct guides our minds away from Rationalization is just one variety of representational exchange: rationally deliberating the possibility of marrying our siblings the process of translating information from one psychological sys- (Lieberman et al. 2007); habit guides our minds away from the tem, or representational format, into another. And representa- possibility of making coffee by putting bread in the toaster tional exchange is useful for the whole organism because it (Morris & Cushman 2017); norms guide our minds away from organizes information in useful ways – ones that best meet its the possibility of catching a ride to the airport by stealing a car demands when the information is required. For instance, some (Phillips & Cushman 2017). Non-rational processes also structure ways of representing information demand little computation but tractable planning by identifying valuable end states (Keramati are relatively inflexible, getting it right in only a restricted range et al. 2016) or goals (Cushman & Morris 2015). A person may of cases. Others require greater computational demands but are seek revenge instinctually, and yet plot his revenge by reasoning; more flexible, getting it right in a wider range of cases. he may seek cocaine habitually, and yet plan to get cocaine by rea- Representational exchange allows an organism to transform rep- soning; he may seek to divide his resources fairly due to blind resentations of one kind into representations of another, making norm adherence, but then reason carefully about how the fairest thought more efficient by balancing the demands of computa- division could be accomplished. tional effort and flexibility. This more general perspective, a the- Thus, even when our behavior is jointly determined by the ory of representational exchange, unifies rationalization with influence of rational and non-rational processes, there is an many other cognitive operations. opportunity for rationalization to extract useful information from the influence of non-rational processes and translate these 4.1. The structure and function of representational exchange into a form useful to the rational system. During rationalization, information flows from non-rational sys- tems to rational ones. Could information flow in the opposite direction – from reason to other adaptive systems, or among 3.8. Summary: Rationalization is rational the other systems themselves (Fig. 4)? Several examples come to Instincts, norms, and habits shape our behavior in adaptive ways, mind. During habitization, choices that were effortful (i.e., ratio- but not by rational planning based on beliefs or desires. Still, these nally planned) become automatic (i.e., habitual). During norm influences are adaptive: We instinctively recoil from precipices internalization, actions that we observed others perform shape because they are dangerous; norm-based food taboos reflect real our intrinsic preferences. Although traditionally these processes toxins, and habitually flipping a light switch is usually a good have been considered unrelated, they are all forms of representa- idea. Rationalization, then, is a useful fiction: When we observe tional exchange: the sharing of information between distinct our own behavior, we infer the beliefs and desires that would mechanisms of behavioral control. have been most likely to have caused that behavior, as if it had Representational exchange is useful because distinct mecha- been an exclusive product of reasoning. Then we adopt those nisms of behavioral control have different ways of representing beliefs and desires. This is adaptive because, on average, the information and guiding action, each with unique advantages new beliefs are true and the new desires promote fitness. For and disadvantages. For instance, habits enable rapid, computa- the same reason, theory of mind might often entail a useful fiction tionally frugal decision making that is occasionally suboptimal, as well: By assuming that others’ behaviors are rational (when they whereas planning attains greater optimality at the expense of are merely adaptive), we can extract useful information. Crucially, time and effort. Representational exchange allows us to keep whether rationalizing our own action or that of others, the process thought efficient – that is, to attain the most important opportu- of inferring information from behavior is structured as a rational nities for flexibility and generalization, subject to the resource inference (Baker et al. 2009). constraint of a limited cognitive capacity. In this manner it fosters In sum, rationalization exchanges representations of Do this! “resource-rational” cognition (Griffiths et al. 2015), improving the for representations of the type Believe this! or Desire that! This overall welfare of the organism. is a kind of representational exchange: It extracts information Consider a simple example. For most people, computing 26 + implicit in the representations of non-rational systems and trans- 52 takes moment of thought, while 25 + 25 comes easily to mind. forms it into the format useful to the rational system. The final This reflects two different cognitive organizations. One system section expands this view of representational exchange, showing encodes a procedure for addition and requires effort to derive spe- how rationalization is one example of a much broader class of cific sums (e.g., 26 and 52). Another system encodes a https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  10 Cushman: Rationalization is rational Some forms of representational exchange will pack informa- tion into compressed forms, storing outputs, abstractions, and heuristics in place of procedures, specifics, and computations. Other will perform the reverse operation, unpacking information by inferring the more detailed and precise information implicit in outputs, abstractions, and rules. Viewed from this perspective, rationalization is a particular kind of unpacking that occurs in the specific context of choice behavior: Specifically, it unpacks behavior into beliefs and desires. It belongs to a broader family of cognitive operations that facilitate representational exchange – not just from non-rational systems to rational ones, but also among the many systems that contribute to decision making. Representational exchange can be situated within a broader taxonomy of operations demanded by a successful multisystem cognitive architecture: 1. Control. What are the several mechanisms that guide our behavior? What representations and computations do they rely on, and what are the distinctive advantages and disadvan- tages of each? Much prior research addresses these questions (reviewed in Dolan & Dayan 2013; Kahneman 2011b; Figure 4. (a) Rationalization is one example of the more general process of (b) rep- Sloman 1996; Squire 2004). resentational exchange. 2. Metacontrol. Which system, or weighted combination, guides our behavior at any given time? In other words, from moment to moment, how do we decide whether to act habitually, ratio- nally, by instinct, or another way? A growing body of contem- precompiled set of sums – roughly, a table in which one looks up porary research addresses this question (e.g., Kool et al. 2017; the entry “25 + 25” and retrieves “50.” The first requires compu- Daw et al. 2005; Griffiths et al. 2015; Shenhav et al. 2013). tation; the second merely requires retrieval. 3. Exchange. What mechanisms enable the exchange of informa- Why do we have two such systems, and why are certain sums tion between systems (Gershman et al. 2014; Lombrozo 2017)? represented one way and other sums another? On the one hand, For instance, how can a behavior formerly produced by rea- knowing the rules of addition is useful because it compresses an soning become habitual, and how can a behavior that was for- infinitely large mapping of inputs to outputs (i.e., arbitrary sets of merly habitual influence subsequent reasoning? This is our numbers to their sums) via a compact rule. This requires far less present focus. memory than, for instance, storing a table of precomputed sums. 4. Exchange control. How do we decide what, and when, to Although effort is required to compute each sum, this is a worth- exchange? Assuming that representational exchange can be while trade-off as compared with the storage demands of the tabu- beneficial in the long run, but also carries immediate costs, lar representation and the learning demands of acquiring it. how is cost-benefit analysis performed? And, when systems On the other hand, certain sums must be computed far more embody conflicting information, which system gets priori- often than others. If you are a cashier who makes change every tized? These are important issues for further development, day, then you do store at least a small table of common sums: but they are not pursued here. “nickel + 2 dimes = quarter,” “4 quarters = dollar,” and so on. These sums are required so frequently that it would be inefficient The next few sections present an account of representational to compute them anew each time. Instead, it is worth storing a exchange somewhat more formally, drawing connections to cur- small cache of common sums for ready and quick retrieval. rent computational models of decision making used in psychol- Any resource-rational cognitive system must find efficient ogy, neuroscience, and computer science. Although the main ways to represent information, managing the competing demands focus is on representational exchange among decision-making of computational effort, memory, accuracy, and flexibility systems, it is clear that the concept applies beyond the domain (Batchelder & Alexander 2012; Griffiths et al. 2015). We must of decision making, and some examples are noted at the end of choose whether to represent procedures or merely their outputs this article. (Gershman et al. 2014; Sutton 1991). We must choose when to represent specifics and when, instead, to fall back on generalities 4.2. The purpose of representational exchange (O’Donnell 2015). We must choose when to be exact and when to satisfied with an approximation (Daw et al. 2005; Gigerenzer & The purpose of representational exchange is to make an organism Selten 2002; Kahneman 2011b; Kool et al. 2017). more biologically fit by making its decision making more effi- It must be rare that we have attained the optimal balance at cient. Efficiency is a balance of accuracy and effort. Thus, some- any given time – and just as rare that the optimal balance could times we increase efficiency by making more accurate decisions; ever be permanent. Rather, as we learn and change – and as other times, by making decisions faster or with fewer cognitive our circumstances and the world around us change – there is a resources. Efficiency can be optimized by sharing information continual demand to adjust the format of the representations across decision-making systems in order to give an individual an that guide our action. This requires mechanisms for representa- array of options: more controlled and accurate thought, or more tional exchange. rapid and automatic thought, depending on the circumstances. https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  Cushman: Rationalization is rational 11 To describe representational exchange in more detail it is use- ful to use some formal concepts and notations. These should highlight useful themes for those who are already familiar with them, but without frustrating those who are not. The purpose is not to offer a formal model of representational exchange, which is well beyond the scope of this article. Rather, it is to establish points of contact with formal models of control and metacontrol developed elsewhere. We envision an organism’s life as a kind of Markov decision process. This means that the organism experiences certain states of the world, and in each of these states, it performs some actions. These actions help to determine the next states it experiences, all of which influence its biological fitness. Any individual’s mind can thus be characterized by the probabilistic mapping from Figure 5. An idealized model of behavioral control. states to actions, or policy. Colloquially, a policy says: “Here is the thing to do in any given situation” (or “the several things you might do and their associated probabilities”). From the reasoning) are useful precisely because they allow an organism standpoint of natural selection, some optimal policy exists that to adjust its policy toward fitness maximization more rapidly – maximizes expected biological fitness. Nobody actually has an on the timescale of a single organism’s life, rather than a multi- optimal policy, but it is a useful ideal to consider: the total set generational one. of instructions for life that maximize your chances of biologically fit children. The closer an organism gets to this ideal, the more fit it is. 4.3.2. Instrumental learning As a simplifying assumption, suppose that instinct, norm com- Whereas instincts implement an innately encoded policy, an indi- pliance, habit, and planning each dictate their own specific policy vidual using instrumental learning instead learns a policy by to an organism. In other words, instincts would provide you with attempting to maximize innately specified rewards. Because one set of instructions; habits with another set of instructions, and instrumental learning occurs within an individual’s lifetime, it so on. These are different mechanisms of behavioral control. In a can improve the individual’s policy faster than natural selection. perfect world every one of these policies would be identical; spe- For instrumental learning to improve an organism’s fitness, natu- cifically, they would all encode the optimal policy. In reality, how- ral selection must assign reward to states or actions that reliably ever, different systems are likely to do better or worse in different increase fitness: things like consuming food, acquiring resources, cases – that is, to recommend more or less fitness-maximizing reproducing, and the like. This often occurs by estimating the actions in different states. Metacontrol is the problem of deciding instrumental value of certain actions – that is, their expected long- how to allocate control to one policy or another in any given sit- run rewards. Current theories of instrumental control tend to use uation, or how to blend them. one of two basic ways of estimating value: habit or planning. The goal of representational exchange is to improve the indi- Habit. Habit learning is often modeled as a method of estimat- vidual policies of each system by transferring information ing the value of every action in every state based on its history of between them. This can improve the overall efficiency of decision reinforcement (Daw & Doya 2006; Sutton & Barto 1998). The making by allowing optimal-but-effortful thought when appropri- major advantage of habit learning is its low computational ate, and suboptimal-but-easy thought when appropriate. demand. First, it only bothers to estimate the value of states and actions that it has actually experienced; for many tasks, this means that the vast majority of conceivable states and actions 4.3. Advantages and disadvantages of control mechanisms are ignored. Second, it precompiles (or caches) the instrumental Before asking how these different influences on our behavior value of actions at the time they are performed, and then draws might exchange information, greater precision is required on upon this cached value representation when making future deci- two points: the different formats in which information is repre- sions. (This is akin to caching the solution to 25 + 25). sented, and the relative advantages and disadvantages of each for- Planning. Planning, like habit, is a variety of instrumental con- mat. Briefly addressing these issues will put us in a better position trol (Daw & Dayan 2014). It estimates the value of actions pro- to understand how and why information might be exchanged spectively, according to the magnitude and probability of between representational formats. reward of their likely outcomes. When this involves searching over a large model of the potential outcomes it is computationally 4.3.1. Instinct demanding. Deriving value estimates from an internal model has Instincts are innate mappings from states to actions that emerge the advantage, however, of making planning flexible. It can sim- regularly in typical development. The advantages of instinct are ulate the outcomes of actions it has never performed, it can speed and reliability: They only depend on the development of update its value estimates based on new information, and it can the organism and not on learning or reasoning, which both also update them based on new specifications of reward. This take time and are contingent upon unreliable experience. If an may be useful when the agent is tasked with planning toward a organism innately possessed a set of instincts comprising the opti- specific goal, for instance because of a hierarchical task decompo- mal policy, it would have no need for learning. In reality, however, sition (Botvinick 2008; Botvinick & Weinstein 2014; Cushman & instincts will not encode the optimal policy because the world Morris 2015; Sutton et al. 1999) or due to social coordination changes too fast for biological natural selection to keep pace. such as joint intentionality (Ho et al. 2016; Kleiman-Weiner Other systems of behavioral control (norms, habits, and et al. 2016; Tomasello 2014). https://doi.org/10.1017/S0140525X19001730 Published online by Cambridge University Press  12 Cushman: Rationalization is rational 4.3.3. Norms and social learning rewards has been observed, the AI can copy human performance Norms are influences on behavior that are learned from others.9 by maximizing those rewards itself, or it can predict human An extensive literature shows when and why social learning is valu- behavior by computing which actions would be reward- able (reviewed in Richerson & Boyd 2008). The basic premise is sim- maximizing for the human. ple enough: Because other people are designed to maximize fitness IRL is easier said than done. In practice, it is often accom- and the things that improve their fitness will often also improve plished by some approximation of Bayesian inference. To see yours, you can improve your own fitness by copying others. how this works, note that we can consider reinforcement learning itself as a probabilistic generative model. That is, given some spec- ification of reward and an environment (composed of many 4.4. Varieties of representational exchange states, actions, and the transition probabilities between them), Having reviewed the representational format of several different reinforcement learning algorithms generate a probability distribu- influences on our behavior and the advantages and disadvantages tion over actions – that is, the policy: of each, we can now consider several mechanisms of representa- tional exchange in greater detail. P(action|reward, environment, state) 4.4.1. Rationalization as inverse reinforcement learning What IRL seeks, however, is the opposite: the probability of We have already seen that rationalization translates observed different rewards and environments given an observed action in actions into useful beliefs and desires. Our next goal is to re- a given state. This can be computed by inverting the generative describe this idea both more formally and more abstractly, reveal- model according to Bayes’ rule (for brevity, action, state, reward, ing useful connections to several literatures. and environment are now represented by their first letters): Natural selection and instrumental learning share a common structure: Both are trying to maximize some objective (fitness P(r, e | a, s) / P(a | s, r, e)P(r, e | s) or reward) by shaping the actions we take in the environments we encounter. An interesting property of rationalization is that The leftmost term states what we want: inferences about it can use the common notion of objective to turn one kind of rewards and environments generated by the observation of what objective (fitness) into another (reward). To see this more clearly, a person does, a, in some state, s. This is proportional to two we will begin by representing both processes (nat