Attention and Prediction Error Learning PDF

Pixabay Prediction Error Learning Rescorla Wagner Model Amygdala and Aversive Prediction Error Dopamine and Reward Prediction Error E. Likhtik Attention in the Reward Prediction Model November 20, 2024 Attent...

Pixabay Prediction Error Learning Rescorla Wagner Model Amygdala and Aversive Prediction Error Dopamine and Reward Prediction Error E. Likhtik Attention in the Reward Prediction Model November 20, 2024 Attention and the Parietal Cortex How can we understand a complex cognitive system? Computational Goal What is the brain system trying to accomplish? Algorithm Computations and representations the brain system uses to facilitate the computational goal Implementation How the structure and function of nervous tissue implements the algorithm To understand a brain system at these three levels, is “to understand [the system] completely” - David Marr (1982)  The Brain as a Prediction Machine Action, perception, cognition arise as organisms minimize the difference between expectation and reality. Classic Cognitive View Predictive Processing Model Behavior Controlled by Separate Mechanisms Behavior is explained by prediction updates Yon et al., Nature Neuro, 2020  The Brain as a Prediction Machine Action, perception, cognition arise as organisms minimize the difference between expectation and reality. If you believe your hypothesis If you believe incoming evidences is more precise than incoming is more precise than initial hypothesis, evidence, hypothesis adjustment is low hypothesis adjustment is higher Yon et al., Nature Neuro, 2020 Non-declarative Memory Model: Learning via Associativity Classical Conditioning: An associative learning process in which an unconditioned stimulus (US) automatically produces the unconditioned response (UR). After repeatedly pairing a neutral, conditioned stimulus (CS) with the US, the CS eventually comes to elicit a conditioned response (CR), often similar to the UR, on its own Prior to conditioning Neutral stimulus (Orientation to sound, no (tone) salivation) US UR (food powder in mouth) (salivation)  Learning via Associativity: Pavlovian Conditioning Classical Conditioning: An associative learning process in which an unconditioned stimulus (US) automatically produces the unconditioned response (UR). After repeatedly pairing a neutral, conditioned stimulus (CS) with the US, the CS eventually comes to elicit a conditioned response (CR), often similar to the UR, on its own Prior to conditioning Neutral stimulus (Orientation to sound, no (tone) salivation) US UR (food powder in mouth) (salivation) Conditioning Neutral stimulus CS (tone) CR (salivation) US (food powder) Non-declarative Memory Model: Learning via Associativity Classical Conditioning: An associative learning process in which an unconditioned stimulus (US) automatically produces the unconditioned response (UR). After repeatedly pairing a neutral, conditioned stimulus (CS) with the US, the CS eventually comes to elicit a conditioned response (CR), often similar to the UR, on its own Prior to conditioning Neutral stimulus (Orientation to sound, no (tone) salivation) US UR (food powder in mouth) (salivation) Conditioning Neutral stimulus CS (tone) CR (salivation) US (food powder) After conditioning CS CR (salivation) CS is Predictive (tone) What about when you don’t learn despite associativity? CS – US association, contiguity CR : You get a glass – Kamin’s Blocking (1969): violates the notion of stimulus association CR : You go get a glass – CR : You go get a glass – Phase III CSB ? Kamin’s Blocking of Associative Learning Simple demo of blocking: https://www.youtube.com/watch?v=CPBV6l4SXXw  Blocking is explained by Prediction Error Experimental Group Control Group Phase I – Phase II – – Phase III Phase III You don’t do anything CSB CR : You get a glass CSB CSB – US association is blocked by learning about CSE – US Prediction Error (PE) – Learning occurs when there is a mismatch between prediction & outcome  Blocking is explained by Prediction Error Learning to read by Pictures + Text: A form of blocking I. II. III. HORSE HORSE  Rescorla – Wagner Model of Associative Learning (1972) Reward Prediction Error = Received reward – predicted reward ∆𝑉𝑉 = 𝛼𝛼𝛼𝛼(λ − 𝑉𝑉) ∆𝑉𝑉 = PE (Suprise!) 𝛼𝛼𝛼𝛼 = Learning rate (𝛼𝛼=salience of CS, 𝛽𝛽 = salience of US) λ = Outcome received (US) 𝑉𝑉 = Outcome predicted by all available cues (CSs) Learning: When outcome received is different that outcome predicted by all available cues, high prediction error No Learning: When outcome received is same as the outcome predicted by all available cues, low prediction error Greeve et al., 2017 Prediction Error (PE) accounts for positive and negative error based learning Rescorla – Wagner Model of Associative Learning (1972) ∆𝑉𝑉 = 𝛼𝛼𝛼𝛼(λ − 𝑉𝑉) Unexpected Reward is Positively Surprising: Positive Reward Prediction Error = Received reward > predicted reward Outcome expected by all CSs US received ( λ) 𝑉𝑉 +𝑃𝑃𝑃𝑃 > Rescorla – Wagner Model of Associative Learning (1972) ∆𝑉𝑉 = 𝛼𝛼𝛼𝛼(λ − 𝑉𝑉) Unexpected Omission of Reward is Negatively Surprising: Negative Reward Prediction Error = Received reward < predicted reward Outcome expected US received ( λ) by all CSs 𝑉𝑉 < −𝑷𝑷𝑷𝑷  Rescorla – Wagner Model of Associative Learning (1972) PE (Suprise!) ∆𝑉𝑉 = 𝛼𝛼𝛼𝛼(𝝀𝝀 − 𝑉𝑉) the US Learning is driven by surprise due to the nature of the US PE explains blocking: The bell (CSe) fully accounts for the US (wine) in Phase I and Phase II. No PE, no learning about the light (CSb) – – Phase III CSB  Many ways to violate expectations Unexpected reward, Omitted punishment Vary expectation while keeping other parameters of the reward constant Example: Vary amount and delay (both are unexpected reward) Calu et al., Neuron (2010)  Ventral Tegmental Area dopamine neurons signal reward and Reward Prediction Error Therapeutic use of reward prediction error Papalini et al., Translational Psychiatry (2020) Dopamine neurons signal reward and reward Prediction Error Wolfram Schultz (1990s), currently in Cambridge. Extracellular recordings in the Ventral Tegmental Area (DA cells) DA cells increase firing to Unexpected Reward (Reward is better than predicted) DA cells fire to CS that predicts reward, but not at time R because the reward here is fully predicted. DA cells decrease firing when a predicted reward is omitted (Reward is worse than predicted) Schultz et al., Science (1997) R-W Model & DA firing explanation of negative PE learning The US reward wasn’t delivered=0 Negative PE e.g. 0-2=-2 ∆𝑉𝑉 = 𝛼𝛼𝛼𝛼(λ − 𝑉𝑉) The CS predicted a reward = a positive value (e.g. 2) Negative PE, DA neuron is inhibited  Possible Teaching Signal for VTA Prediction Error DA cells Lateral Habenula is a midbrain structure with glutamatergic (excitatory) projections to VTA Reward Omission Unexpected Reward Matsumoto & Hikosaka, Nature (2007) Possible Teaching Signal for VTA Prediction Error signaling Lateral Habenula neurons: increase firing to No Reward decrease firing to Reward have shorter latencies than DA VTA neurons stimulation, inhibits VTA DA neural firing Matsumoto & Hikosaka, Nature (2007) Model for lateral habenula as a teaching signal of DA VTA neurons No reward Excitation of inhibitory No reward neurons LH: + VTA: - DA Glut GABA Matsumoto & Hikosaka, Nature (2007) Hong et al., J Neurosci (2011) Ji & Shephard, J Neurosci (2007) Eshel et al., Nature (2015)  Prediction Error Learning and DA in humans Cognitive Decline in PD patients: - memory failure - trouble with attention/ concentration - slower associative learning (adapting to new CS-US contingencies) - Prediction Error failure Activity in Striatum of basal ganglia Pessiglione et al, Nature (2006)  Prediction Error Learning and DA in humans Prediction Error based learning depends on contiguity (close association of CS and US in time). Violation of this relationship during short delays between CS-US will affect PD patients more than healthy controls. Foerde & Shohamy, JNS, 2011  Striatum is more active with immediate feedback leanring Foerde & Shohamy, JNS, 2011  PD patients are worse at learning with immediate feedback Foerde & Shohamy, JNS, 2011 The Amygdala and Unexpected Punishment Unexpected punishment is also a type prediction error learning Unexpected punishment Pleasant Tone (CS+) Shock to Hand + Amygdala activation to unexpected punishment Tissue Oxygenation (TO2) 1st time 2nd time 3rd time McHugh et al., JNS (2014)  The Amygdala and Unexpected Punishment The Amygdala shows a greater response to Unexpected shocks than expected shocks McHugh et al., JNS (2014) The amygdala is a detector of the unpredictable Amygdala neurons fire to the unpredictable sequence Herry et al., JNS (2007) The amygdala is a detector of the unpredictable Herry et al., JNS (2007)  Attention in learning Reward Prediction Error = Received reward – predicted reward ∆𝑉𝑉 = 𝛼𝛼𝛼𝛼(λ − 𝑉𝑉) ∆𝑉𝑉 = PE (Suprise!) 𝛼𝛼𝛼𝛼 = Learning rate (𝛼𝛼=salience of CS, 𝛽𝛽 = salience of US) λ = Outcome received (US) 𝑉𝑉 = Outcome predicted by all available cues (CSs) Salience affects how much attention is given to a stimulus In R-W model of reward prediction, salience adjusts learning rates. Other learning models include attention as well. Attention is an early processing module. It allows for selective focus to stimuli.  Endogenous vs Exogenous attention Endogenous attention – internally generated spotlight, does not depend on looking at the stimulus. Increases reaction time. Exogenous attention – externally generated orienting to environmentally generated stimuli. Loud sounds, etc.  Endogenous Attention The Cocktail Party Effect https://www.youtube.com/watch?v=mN--nV61gDo This film was made by Mark Mitton and Josh Aviner, with help from Susanna Mitton. Speaker-Listener synchrony during attended speech The Cocktail Party Effect Brain – to – brain synchrony in listener and attended speaker, at the temporal –parietal junction, even before the listener verbally responds to speaker. Listener Dai et al., Nat Comm., 2018  Endogenous & Exogenous Attention https://www.youtube.com/watch?v=U1saQoMRD8A Parietal cortex plays a role in attention Right Parietal cortex in visual attention Right parietal cortex is active when attending left visual field Right & Left Parietal cortices are active when attending L&R visual field Right Parietal cortex in visual attention Visual Hemi-Neglect Injury in the right parietal lobe results in the loss of attention to the left visual field. The visual cortices are spared, thus objects are seen, but not attended Tend to ignore the left side of one’s own face (eg. when shaving or applying make up).