cogpsy_slides_merged.pdf

Cognitive Psychology and its Applications Joshua Snell [email protected] Today Practicalities: - Location, Study materials, Canvas - Schedule - Examination Course objectives Research topics A little research exercise About myself 2012: Bachelor’s degree in Utrecht Liberal Arts & Sciences 2014: Mas...

Cognitive Psychology and its Applications Joshua Snell [email protected] Today Practicalities: - Location, Study materials, Canvas - Schedule - Examination Course objectives Research topics A little research exercise About myself 2012: Bachelor’s degree in Utrecht Liberal Arts & Sciences 2014: Master’s at the VU Cognitive Neuroscience 2015-2018: PhD in Marseille Researching reading Practicalities Monday 11:00 → HG10A00 Thursday 15:30 → HG02A00 Study materials: An introduction to human factors engineering (Wickens et al.) + Canvas material Disclaimer New course design ! Less focus on learning human factor principles; more focus on knowledge of the brain and designing applied cognitive science experiments Cognitive psychology: what is it? The study of the (human) brain and behavior Understanding brain and behavior in terms of its functions (cognitive processes): - perception, attention, memory, motor control, executive functions Schedule 3 components: - lectures - workshops - research Examination Exam (October 25th): 50% Research project: 30% Workshop assignments: 10% Participation: 10% On each component you have to score >5.4 Course objectives - Become savvy on the brain and behavior - Get experience with doing research (design and build experiments, collect and analyze data) - Learn to translate theory into experiment - Learn to report your science Meaning: you’ll become an (applied) scientist Research project In groups of ~5, you’ll design, build, and carry out a study in a topic of your interest Various applied topics OpenSesame software (Python-based) Lab space available: MF building 4th floor Topics 1) Banknote design & Counterfeit detection 2) Reading & dyslexia 3) Beach flag design & safety 4) Horizontals vs. verticals in fashion 5) Intuitive roads 6) The least interfering halo 7) Salient teammates In groups of 5: Brainstorm about an experiment for investigating either a role of attention or memory Think about conditions Measures of interest Your predictions Canvas In groups of 5: Post your idea on Discussion page before Sept 7th Brainstorm about an experiment for investigating either a role of attention or memory This Thursday: No meeting We’ll form teams and you’ll receive a Think about conditions briefing For next week: install OpenSesame from osdoc.cogsci.nl Measures of interest Your predictions From theory to prediction to experiment + OpenSesame workshop Why talk about theory? Methods and techniques aren’t our only instruments; theories are instruments too! Methods and techniques dictate how we investigate things... Theory dictates what we investigate and why Do we always need theory? No. Purely descriptive / observational research can be useful too. Do we always need theory? Ultimately science is not just about knowing every thing, but about discovering the laws that connect all things → Not just what, but also why Laws are captured in theory What is a good theory? Explanatory scope Parsimony A theory must be falsifiable If a theory cannot be tested, we have zero knowledge about its plausibility A theory must be falsifiable Example: serpent churches? Sigmund Freud: Psycho-analysis Testing a theory in cognitive psychology “A causes B” “B relies on A” “The influence of A on B relies on C” The cognitive psychologist’s challenge: How can we manipulate A and measure B? Testing a theory in cognitive psychology “Expectations influence perception” “B relies on A” “The influence of A on B relies on C” The cognitive psychologist’s challenge: How can we manipulate ‘expectations’ and measure ‘perception’? Testing a theory in cognitive psychology “Expectations influence perception” “B relies on A” We need experimental conditions e.g., one with ‘high expectation’, the other with ‘low expectation’ We need to devise a behavioral measure of perception e.g., an indication by a participant whether a visual object is blue or yellow Testing a theory in cognitive psychology We have to make everything quantifiable → we must be able to translate everything into numbers “B relies on A” Do we take other factors into account? “But wait, older people are more sensitive to yellow than to blue” “But wait, when the sky outside is blue, this may influence responses” Do we take other factors into account? Within-subjects design: each subject is tested in all conditions Between-subjects design: different subjects are tested in each condition We typically always prefer a within-subjects design over a between-subjects design, as it allows us to ignore factors that potentially have an influence The replication crisis in psychology The replication crisis in psychology Approximately 40% of published studies could be replicated... might 60% of the science out there be erroneous? The replication crisis in psychology Various causes - publication bias “file drawer problem” - Bad practices – intentional and unintentional! - Overall pressure to produce ‘sexy’ results The replication crisis in psychology How can we have more confidence in our results? → Have abundant statistical power → Replicate our own experiments → Engage in theory-driven research; preferably attempting to falsify a theory rather than to confirm it The replication crisis in psychology Statistical power → the chance that an effect is established, given that the hypothesis is true Brysbaert & Stevens (2018): 1,600 measurements per condition (e.g., 25 subjects, 64 trials per condition) Testing theories without experiments →Computational models We can test the extent to which a computer model can accurately simulate behavior. The more accurately it simulates behavior, the more support we have for each of the model’s assumptions. Testing theories without experiments →Computational models Summary - Theories are instruments, just like methods and techniques. Methods: how. Theory: what & why - Without theories we can still make valuable observations. But theories are the glue that should hold everything together. - The 3 characteristics of a good theory: Explanatory scope, Parsimony, and Falsifiability - Don’t hold venomous snakes. - H: A → B. Our mission: devise ways to manipulate A, and to measure (quantify) B. - When possible, stick to a within-subjects design Summary - For a future without replication crisis: Have abundant statistical power, replicate our own experiments, aim to falsify theories - Approx. 1,600 measurements per condition should do. - Testing theory without experiment: computational model. Model outputs behavior, behavior is compared to human behavior; the more resemblance, the more support for model assumptions. Perception & Attention Joshua Snell [email protected] Cats, Dogs & Capybara’s - shortcomings? Multisensory Integration Perception We will cover: - What is perception, and what is sensation? The core challenge: resolving ambiguity Bottom-up versus top-down processes Bottom-up: Gestalt principles Top-down: Experience Neurophysiology What is perception? Sensation: The registration of a physical stimulus by receptive neurons Example: activation of olfactory bulb What is perception? Sensation: A physical, factual thing, not susceptible to interpretation etc. Example: activation of olfactory bulb What is perception? Sensation: A physical, factual thing, not susceptible to interpretation etc. Example: activation of visual cortex What is perception? Fovea (1°): many cones, sharp vision (high acuity), color vision, lower sensitivity Parafovea (6-8°): mix of cones and rods Perifovea (>8°): mostly rods, low acuity, no color vision, more sensitivity What is perception? Perception: the process of interpreting sensations “Stink” Example: Smelling What is perception? The goal: Interpreting, recognizing, understanding (‘what is it?’) Interacting with the world (‘How to respond?’) A thin line between sensation and perception... Pupillary light response is not just triggered by incoming light... but also by thinking about bright objects! “Sun” “Darkness” A thin line between sensation and perception... Pupillary light response is not just triggered by incoming light... but also by thinking about bright objects! The core challenge in perception: to resolve ambiguity The inverse projection problem → From sensory processing alone we cannot say anything conclusive about the world! The core challenge in perception: to resolve ambiguity The inverse projection problem → From sensory processing alone we cannot say anything conclusive about the world! The core challenge in perception: to resolve ambiguity An auditory example... Green needle Brainstorm Bottom-up versus top-down processing bottom-up: Sensory organs provide activation of ‘low’ cortical regions, cascades to ‘higher’ regions Bottom-up versus top-down processing bottom-up: Sensory organs provide activation of ‘low’ cortical regions, cascades to ‘higher’ regions Bottom-up versus top-down processing bottom-up: Sensory organs provide activation of ‘low’ cortical regions, cascades to ‘higher’ regions Bottom-up versus top-down processing bottom-up: Sensory organs provide activation of ‘low’ cortical regions, cascades to ‘higher’ regions Bottom-up versus top-down processing bottom-up: Sensory organs provide activation of ‘low’ cortical regions, cascades to ‘higher’ regions Bottom-up versus top-down processing Top-down: ‘higher’ regions influence activation of ‘lower’ regions Bottom-up versus top-down processing sensation = bottom-up perception = mixture of both Bottom-up processing: perceptual organization Let’s see why bottom-up processing might be not just sensation... Grouping of local features into global structures seems to proceed automatically Bottom-up processing continuity Gestalt principles: A set of assumptions about things that happen in an automatic, bottom-up fashion ...a mere product of the system’s architecture similarity proximity closure symmetry common fate and one that relies on motion → Bottom-up processing Gestalt principles: A set of assumptions about things that happen in an automatic, bottom-up fashion similarity, proximity, symmetry, closure, continuity, common fate Bottom-up processing Gestalt principles: A set of assumptions about things that happen in an automatic, bottom-up fashion ...a mere product of the system’s architecture But are all these ‘effects’ really the result of bottom-up processes? Probably not. Our life experiences bolster the expectation that - Similar-looking things belong together - Objects are most often symmetrical Top-down processing This is how we typically conceptualize processing in the brain. - various levels of processing - interactions among levels “Sun” green needle vs. brainstorm Top-down processing James McKeen Cattell, 1886 The word-superiority effect: a letter is recognized faster if if it is in a word than if it is in a non-sensical string PLUMP PMULP Another way to frame the interaction between top-down & bottom-up Predictive coding A grand unifying theory of the mind? Neurophysiology: perception in the brain BUT: it’s never about isolated brain area’s Neurophysiology: Dorsal and Ventral pathways ‘Dorsal’ = backside: ‘Where’ pathway ‘Ventral’ = belly: ‘What’ pathway Plasticity in the brain: the brain is flexible Where do ‘detectors’ come from? Our experiences shape dedicated clusters of neurons The sad-cat story Recap Sensation vs. perception: a thin line Perception is about interpreting and interacting with the world Bottom-up & top-down processes In vision: from back of brain to front → from lower to higher levels of cognition Neuronal plasticity Attention - What is attention? - Various types of attention: Spatial vs. feature-based attention Top-down vs. bottom-up attention Endogenous vs. exogenous attention Overt vs. covert attention - Early or late selection? - Is attention the key to everything? → Feature Integration theory - Attentional disorders What is attention? Many psychologists have provided definitions... Here is one: Attention is the mind’s capacity to enhance and suppress sensory input and internal representations Also applies to things that we keep in memory Why do we need attention? Tap with your hands on your knees: left left right right left left right right etc. super easy! Now simultaneously count backwards from 100 in steps of 3 100, 97, 94, 91, 88 etc. How’s the tapping going now? Why do we need attention? Our brain cannot do an infinite number of computations at once both consciously and subconsciously Computations must be run to completion at the expense of other computations both consciously and subconsciously Let’s discuss various types of attention... Within the realm of vision: Overt & Covert attention Overt is Obvious to others; the eyes and head move Covert is Concealed to others; the eyes and head do not move In the lab we often track overt attention How might we track covert attention? Let’s discuss various types of attention... Within the realm of vision: Overt & Covert attention Mathôt et al.: The pupil responds to the brightness of covertly attended locations Let’s discuss various types of attention... Within the realm of vision: Spatial vs. Feature-based attention Attentional orienting in vision is often spatial ...but you can choose to be more ‘sensitive’ to apples; we focus in terms of both where and what Let’s discuss various types of attention... Endogenous vs. Exogenous attention internally driven (by ourselves) vs. externally driven (by the world) Endogenous or exogenous? Attention research in the 1950’s Participants heard two messages simultaneously and had to focus on one... ...and could not report what had been said in the other stream endogenous But when hearing one’s own name, attention is captured. exogenous Endogenous or exogenous? Attention research in the 1950’s Participants heard two messages simultaneously and had to focus on one... ...and could not report what had been said in the other stream endogenous But when hearing one’s own name, attention is captured. exogenous → ‘cocktail party effect’ The singleton paradigm Respond to the line orientation in the red circle The singleton paradigm Respond to the line orientation in the red circle So how does all this work, cognitively? Let’s look at neurons... Recall the action potential mechanism we discussed in Lecture 1. Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials So how does all this work, cognitively? Let’s look at neurons... Recall the action potential mechanism we discussed in Lecture 1. Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials So how does all this work, cognitively? Let’s look at neurons... Recall the action potential mechanism we discussed in Lecture 1. Topographic organization of the visual cortex Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials So how does all this work, cognitively? Let’s look at neurons... Recall the action potential mechanism we discussed in Lecture 1. Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials Some connections are inhibitory rather than excitatory. Neurons coding for Hillary’s upper visual field may have suppressed neurons coding for Hillary’s lower visual field when ‘the thing’ happened. So how does all this work, cognitively? Let’s look at neurons... Recall the action potential mechanism we discussed in Lecture 1. Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials Some connections are inhibitory rather than excitatory. Neurons coding for Hillary’s upper visual field may have suppressed neurons coding for Hillary’s lower visual field when ‘the thing’ happened. So how does all this work, cognitively? Let’s look at neurons... Recall the action potential mechanism we discussed in Lecture 1. Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials Some connections are inhibitory rather than excitatory. Neurons coding for Hillary’s upper visual field may have suppressed neurons coding for Hillary’s lower visual field when ‘the thing’ happened. So how does all this work, cognitively? Neurons have thresholds for when to fire The more a neuron is excited (the more input it receives via its dendrites), the more frequently it will fire action potentials Some connections are inhibitory rather than excitatory. Neurons coding for Hillary’s upper visual field may have suppressed neurons coding for Hillary’s lower visual field when ‘the thing’ happened. Signals sent by the upper-visual-field neurons will have entered conscious awareness (frontal brain regions) faster So how does all this work, cognitively? Exogenous attention: strong sensory input tips the balance (in terms of the ‘neuron battle’ described on previous slide) Endogenous attention: higher-order neurons suppress or excite neurons at the level of perception Recall story about bottom-up & topdown interactions Biasing of low-level detectors by higher levels is a form of endogenous attention! Attentional disorders (Hemispatial) neglect In all these tasks, one side (hemifield) is ignored, even though the patient can see things in that hemifield when attention is forcefully directed to it Attentional disorders (Hemispatial) neglect Where is the lesion? In all these tasks, one side (hemifield) is ignored, even though the patient can see things in that hemifield when attention is forcefully directed to it “where” “what” Attentional disorders (Hemispatial) neglect Where is the lesion? In all these tasks, one side (hemifield) is ignored, even though the patient can see things in that hemifield when attention is forcefully directed to it “where” “what” Recap Attention: - is the mind’s capacity to enhance and suppress sensory input and internal representations - exists because the brain can only do so many computations at once Various types of attention: spatial vs. feature-based attention, endogenous vs. exogenous attention, overt vs. covert attention Remember the neural dynamics explanation Hemispatial neglect Research Toolbox Response time, accuracy & signal detection theory Research Toolbox Practical points - Start implementing your experiment - Helpdesk opened on Canvas > Discussions - Recorded lecture Memory & Decisionmaking online tomorrow - Monday 25th: eye-tracking & pupillometry Response times response output Response times Response times What we typically do in processing RTs: - exclude incorrectly answered trials - exclude very atypical trials (i.e., trials with RT beyond several SDs from the mean) From today’s module, download data_21_09.txt and Analyses_lecture_21_09.R Response times From today’s module, download data_21_09.txt and Analyses_lecture_21_09.R Response times Is this all that there is to RTs? Distributions could reveal more information → A difference between two response conditions may be more strongly expressed in the faster portion of RTs than in the slower portion. (Gomez & Perea, 2020) Response times A case study: the Stroop task (Stroop, 1935) Word meaning impacts processing of the word’s print color – and vice versa RED RED BLUE BLUE Response times Let’s check it out ourselves withtask a (simulated) experiment A case study: the Stroop (Stroop, 1935) Participants saw the words RED and BLUE, in red or blue print. Word meaning impacts processing of the In one block, they responded the meaning of the word. In word’s print color – and vice versa another block, they responded the print colour of the word. H: slower responses when the meaning and color don’t match RED RED BLUE BLUE Response times data_21_09.txt Canvas in today’s A case study:onthe Stroop task module (Stroop, 1935) Analyses_lecture_21_09.R in today’s module Word meaning impacts processing of the word’s print color – and vice versa Response times data_21_09.txt Canvas in today’s A case study:onthe Stroop task module (Stroop, 1935) Analyses_lecture_21_09.R in today’s module Word meaning impacts processing of the When calculating the mean response times (RTs) in each condition, we word’s print color – and vice versa indeed see effects of meaning/print congruency: meaning decision congruent: 587 ms incongruent: 611 ms print decision congruent: 489 ms incongruent: 510 ms Response times data_21_09.txt Canvas in today’s A case study:onthe Stroop task module (Stroop, 1935) Analyses_lecture_21_09.R in today’s module Word meaning impacts processing of the Are these two effects the same thing, cognitively speaking? word’s print color – and vice versa → Let’s look at some density plots meaning decision congruent: 587 ms incongruent: 611 ms print decision congruent: 489 ms incongruent: 510 ms Response times Early RTs are very similar between conditions, late RTs differ a lot. ...so this effect has a late temporal locus. Decisions about stimulus color Response times Late RTs are very similar between conditions, early RTs differ a lot ...so this effect has an early temporal locus. Decisions about word meaning Accuracy Analyses of accuracy are often regarded as being interchangeable with analyses of RT. (better performance: shorter RTs and fewer errors) In most behavioral tasks we look at both. Having more measures provides a broader picture. Sometimes we only look at one measure. (e.g., many lines of memory research) Accuracy Is it problematic if we find an effect in accuracy but not in RTs? → No. Persons A and B are equally fast, but A is more accurate: A performed better A and B are equally accurate, but A did it quicker: A performed better. Accuracy Is it problematic if opposite effects are found in accuracy and RT? → Yes. Person A is better at shooting, but person B is better at skiing. We cannot tell who is the better biathlete. Combining RTs and accuracy Inverse efficiency scores (reading material: Bruyer & Brysbaert, 2011) Combining RTs and accuracy into one measure (IES) may allow us to make better direct comparisons. IES = RT / P(correct) RT = 500 ms, accuracy = 0.90 → IES = 500/0.90 = 556 ms. RT = 480 ms, accuracy = 0.80 → IES = 480/0.80 = 600 ms. A deeper look into accuracy Signal detection theory Only applicable in the context of binary decisions A deeper look into accuracy Signal detection theory A more elaborate measure of accuracy: Sensitivity The world around us is noisy → How well can we distinguish the relevant from the irrelevant? Sensitivity doesn’t only look at our ability to spot the relevant, but also at our ability to ignore the irrelevant What is the key challenge in perception? To resolve ambiguity What is the key challenge in perception? To resolve ambiguity; and to distinguish the relevant from the irrelevant Signal Detection Theory: A way to quantify perceptual skill Why do we need to do this? Signal Detection Theory: A way to quantify perceptual skill Why do we need to do this? Signal Detection Theory: A way to quantify perceptual skill Task: push button when detecting an unnatural source of light Signal Detection Theory: A way to quantify perceptual skill BUT... the universe is noisy and so are our senses ? Signal Detection Theory: A way to quantify perceptual skill Who is the better observer now? = hits = false alarms Sound intensity without vs. with alarm NOISE SIGNAL Sound intensity without vs. with alarm NOISE SIGNAL Sound intensity without vs. with alarm NOISE SIGNAL Sound intensity without vs. with alarm No matter where your threshold is, your ability to distinguish signal from noise is the same! Cucumber neuron’s firing rate when seeing a zuccini vs. a cucumber: 4 outcomes correct rejection false miss alarm The response matrix: the proportions of hits, misses, false alarms and correct rejections depend on: - your threshold - the distance between signal and noise distributions The response matrix: the proportions of hits, misses, false alarms and correct rejections depend on: - your threshold - the distance between signal and noise distributions The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) How do we measure this distance? We cannot measure ‘perceived intensity’! ... or can we? The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) How do we measure this distance? We cannot measure ‘perceived intensity’! ... or can we? The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) How do we measure this distance? We cannot measure ‘perceived intensity’! ... or can we? 1.08 The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) How do we measure this distance? We cannot measure ‘perceived intensity’! ... or can we? The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) How do we measure this distance? We cannot measure ‘perceived intensity’! ... or can we? -1.75 The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) Sensitivity = z-score for hits minus z-score for false alarms : 1.08 - -1.75 = 2.83 -1.75 The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) All that we need to measure sensitivity are the proportions The distance between signal and noise distributions varies among individuals and is called sensitivity (= perceptual skill! ) Not affected by response threshold (criterion) ! z(hits) – z(false alarms) remains same Staircase procedures What if we want to measure performance irrespective of these subjective perceptual processes? e.g., a person is slightly color-blind in our Stroop task RED BLUE BLUE RED Staircase procedures a.k.a.: Controlling the subjective distance between the relevant and the irrelevant → adjust stimulus intensity, duration, etc., on the basis of incoming responses → so that all subjects perform equally Staircase procedures “Words are impacted by surrounding words” “What is the developmental trajectory of this?” Standard paradigm: show words for 150 ms “Uh oh, kids can’t even recognize single words in 150 ms” Staircase procedures : example - After X correct trials, decrease stimulus duration by β - After Y incorrect trials, increase stimulus duration by β - After each oscillation, decrease β a bit (until it hits 0) Next Monday Cognitive Psychology Memory & Decision-making Joshua Snell [email protected] Memory is... Any way in which a past experience affects future thoughts or behaviors The Modal Model of Memory – 1968 Today, we still conceptualize various stages of memory Sensory, STM/WM, LTM What’s the difference between sensory- & short term memory (STM)? Think back of lecture 3: sensation vs. perception! Sensation ≈ sensory memory, because neural activity caused by a sensation isn’t turned off like a light switch What’s the difference between sensory- & short term memory (STM)? Think back of lecture 3: sensation vs. perception! Sensation ≈ sensory memory, because neural activity caused by a sensation isn’t turned off like a light switch Activity in early regions decays over time What’s the difference between sensory- & short term memory (STM)? Think back of lecture 3: sensation vs. perception! Sensory memory examples What’s the difference between sensory- & short term memory (STM)? Our senses register a lot of information (e.g. the whole visual field), but only part of it is consciously processed A.K.A. only part of it enters STM (= attentional orienting!) What’s the difference between sensory- & short term memory (STM)? Sperling (1960) What’s the difference between sensory- & short term memory (STM)? Sperling (1960) “Short-lived sensory memory registers all or most of the information that hits our visual receptors, but this information decays within less than a second” What’s the difference between sensory- & short term memory (STM)? Sperling (1960) “Short-lived sensory memory registers all or most of the information that hits our visual receptors, but this information decays within less than a second” What’s the difference between sensory- & short term memory (STM)? STM is the first stage where we can pro-actively retain things STM: what’s the limit? Try to memorize the following sequence 529846731 STM: what’s the limit? Try to memorize the following sequence and the next... 123456789123456789 STM: what’s the limit? 529846731 vs. 123456789123456789 chunking The learned relationships among objects are a matter of long-term memory ... yet, this knowledge does aid STM → interdependence Interaction between top-down & bottom-up (perception lecture) works for memory too! STM: what’s the limit? Instead of framing the limit in terms of number of objects, frame it in terms of amount of information. Some item types are more difficult to remember STM: actively memorizing stuff, and...? Calculate (3^3)/2 This task relied on STM; yet you did more than just memorize. Enter Baddeley & Hitch (1974) Working memory Working Memory “slave systems” Working Memory “slave systems” Working Memory The visuo-spatial sketchpad: a ‘space’ to navigate in – with attention Attention can be moved within WM space → a form of ‘manipulating’ information in WM Freek van Ede (from the VU) Working Memory How does it work in the brain? Sensory memory is easy: residual activity in early perceptual regions of the brain Working Memory Various regions of the brain have been associated with each of these WM components The prefrontal cortex is key WM in the brain WM in the brain PFC is key... but so are all our perceptual areas WM in the brain PFC is key... but so are all our perceptual areas We can ‘read minds’ by looking at the visual cortex So memories are ‘stored’ - or at least read out in the visual cortex too looking → ‘Higher’ regions like the PFC coordinate activation in perceptual regions during retention memorizing Long term memory → what is it? how is it different from STM / WM? What is LTM? LTM is the seemingly infinite archive into which we have stored every experience since our existence What is LTM? Imagine chewing on your desk Taste and smell are hard-to-describe- but very robust memories What is LTM? 8-month old babies depict more attention for familiar words than novel words ...But 20 years later, these humans will probably not remember that they took part in that experiment Procedural / implicit memory vs. episodic memory (statistical learning) What is LTM? - LTM is the seemingly infinite archive into which we have stored every experience since our existence - Though the archive is infinite, stored files may ‘wither’ What is LTM? - LTM is the seemingly infinite archive into which we have stored every experience since our existence - Though the archive is infinite, stored files may ‘wither’ - Throughout our lives, we are automatically building the archive - for strategic purposes learning, automatization, bolstering WM The interaction between WM and LTM First of all: how do we know that these are really two separate things in the brain? The interaction between WM and LTM Clive Wearing: STM ‘alright’, LTM impaired First of all: how do we know that these are really two separate things in the brain? The interaction between WM and LTM Patient K.F.: impaired WM, but LTM intact First of all: how do we know that these are really two separate things in the brain? The interaction between WM and LTM First of all: how do we know that these are really two separate things in the brain? An experiment without patients: The serial position curve briefly presented one-by-one: tree – laptop – sphinx – earbud – mouse – lamp – pocket When asked to recall as many words as possible, subjects report the first and last words best The interaction between WM and LTM First of all: how do we know that these are really two separate things in the brain? An experiment without patients: The serial position curve briefly presented one-by-one: tree – laptop – sphinx – earbud – mouse – lamp – pocket When asked to recall as many words as possible, subjects report the first and last words best The interaction between WM and LTM First of all: how do we know that these are really two separate things in the brain? primacy effect (first word advantage): first words get full attention; STM not occupied by other things, and/or words were rehearsed for a longer amount of time recency effect (last word advantage): Last words are still in STM The interaction between WM and LTM Information in LTM is constantly re-activated by the things that we keep in WM. This information in turn bolsters whatever is kept in WM. - Meaning of words - Relevant past events - Goals Neural mechanisms later The interaction between WM and LTM Information in LTM is constantly re-activated by the things that we keep in WM. This information in turn bolsters whatever is kept in WM. Example: The various types of LTM Episodic Procedural Implicit vs. explicit Semantic The various types of LTM Episodic (explicit) Procedural (implicit) Implicit vs. explicit Semantic (explicit) other word for explicit: declarative What’s the difference between semantic and episodic memory? Past experiences vs. facts ‘mental time travel’ learned relationships What’s the difference between semantic and episodic memory? A double dissociation: two patients... Displayed good knowledge about many things but forgot things that happened 3 minutes prior Didn’t know the meanings of words anymore, didn’t recognize close relatives; but could recount the previous day, week, or year How does LTM work in the brain? STM: prefrontal + perceptual regions LTM: Hippocampus in the medial temporal lobe How does LTM work in the brain? Hippocampus in the medial temporal lobe strongly involved in memory tasks When depriving people of sleep, memories don’t stick as well How does LTM work in the brain? Consolidation How does LTM work in the brain? LTM and STM work similarly in the sense that perceptual regions of the brain are involved (memorizing a visual = ‘simulating’ seeing something) ...but what about implicit learning? ...but what about implicit learning? Learning at the level of single neurons: with repeated activation, there is a chemical change at the synapse. The synaptic transfer is strengthened Ergo, faster processing e.g., stronger connections between letters and words Recap Memory: any way in which a past experience impacts present/future thoughts or behaviors. Several memory stages: sensory memory, STM / WM, LTM Sensory memory = trace (residual) activity; high capacity STM = active maintenance (due to attentional focus); low capacity WM = STM whereby information is not just memorized, but also manipulated Prefrontal cortex is key, but in cadence with perceptual regions (‘slave systems’) Recap LTM: the seemingly infinite archive into which we stow files that may wither over time Close interactions between WM & LTM: LTM is activated by the things kept in WM, and this info in turn bolsters WM content Evidence for separate systems: patients, serial position curve Understand the various types of LTM: semantic, episodic, implicit vs. declarative, procedural, statistical learning / priming Episodic / declarative memory: interaction hippocampus and cortex Implicit learning can also be explained at the level of single neurons Part 2: Decision-making Decision-making First of all: do we make decisions? Free will is an endless debate “Decision” implies multiple options ...But decisions can always be explained → if we’ve completely figured out the brain, can we fully predict human behavior? Decision-making Perception & Action Decision-making is the bridge between perception (+memory, emotions, biases, predispositions, etc. etc. etc.) and action Expected utility theory “Given knowledge about what the outcomes of various options will be, people choose whatever yields maximum value” Not true Confirmation biases and overconfidence biases We give more weight to information that confirms our expectations Confirmation biases and overconfidence biases We trust ourselves more than others 75% of drivers think they belong to the best 25% Decision-making Decision-making Expected utility theory: “having all relevant information, people will make a decision that yields the most utility/value/achievement” Prospect theory: “people act on predicted emotions” How good would I feel if I win? How bad would I feel if I lose? People are often risk-averse; but it also depends on how the problem is framed! Decision-making Framing: when emphasizing gains, people become more risk-averse, but when emphasizing losses, people become more risk-taking A: 200 are saved B: 1/3 chance that 600 are saved, 2/3 chance that none are saved C: 400 will die D: 1/3 chance that none die, 2/3 chance that 600 die Decision-making Framing: when emphasizing gains, people become more risk-averse, but when emphasizing losses, people become more risk-taking A: 200 are saved C: 400 will die B: 1/3 chance that 600 are saved, 2/3 chance that none are saved D: 1/3 chance that none die, 2/3 chance that 600 die Decision-making Judges are animals too Decisions in the brain An ‘unfairness’ region in the brain: the anterior insular cortex Higher activity → less likely to accept an unfair offer “We have 10$, I give you 3$ and keep 7$ myself” The drift diffusion model Two competing neuronal clusters, evidence accumulates until one cluster (representing one decision) reaches threshold Until then: doubt The drift diffusion model Two competing neuronal clusters, evidence accumulates until one cluster (representing one decision) reaches threshold Until then: doubt Neural evidence: in Rhesus monkeys, direction-selective neuronal clusters are activated until one cluster’s spike rate hits threshold The drift diffusion model Does it hold for more complex decisions? Ultimately, to understand decision-making is to understand the brain entirely... We have neuronal clusters driving the onset of billions of actions Those clusters are excited in billions of ways We can predict ‘decisions’ of single neurons and human populations; nothing in between Eye-tracking and pupillometry When and why do we need eye-tracking? Descriptive research: we’re interested in the what, why and where of eye movements themselves Explanatory research: oculomotor data may provide a window onto various cognitive processes Terminology Saccade Saccadic amplitude Saccadic latency Fixation Fixation duration Microsaccade Terminology Saccade Saccadic amplitude Saccadic latency Fixation Fixation duration Microsaccade Eye position: two signals Pupil location Corneal reflection of (infrared) light sent from camera Eye position: calibrate Pupil location Corneal reflection of (infrared) light sent from camera Eye position: calibrate later... Pupil location Corneal reflection of (infrared) light sent from camera In OpenSesame... In OpenSesame... In OpenSesame... In OpenSesame... In OpenSesame... In OpenSesame... Thus far: Tracking the eye position (and pupil size) in a normal behavioral experiment But what about gaze-contingent trickery? From today’s module on Canvas, download eyetracking.osexp Mac users: may have to install PyGaze package manually A variant of the Posner cueing task Attention is biased by top-down cues e.g., it takes longer for you to note the square on the right, when the arrow points to the left. Task: indicate location of square (left / right) Potential confound: Simon effect (Maybe arrow biased response rather than attention) A variant of the Posner cueing task Attention is biased by top-down cues e.g., it takes longer for you to note the square on the right, when the arrow points to the left. Task: Move eyes to the square Potential confound averted A variant of the Posner cueing task Attention is biased by top-down cues e.g., it takes longer for you to note the square on the right, when the arrow points to the left. Task: Move eyes to the square Potential confound averted A variant of the Posner cueing task Attention is biased by top-down cues e.g., it takes longer for you to note the square on the right, when the arrow points to the left. Task: Move eyes to the square Potential confound averted Eye-tracker data In our experiment we have a simple response_time variable... Do we need more? If possible, avoid having to dive into these files → define DV’s in OpenSesame Eye-tracker data → define DV’s in OpenSesame Example: saccadic curvature Eye-tracker data → define DV’s in OpenSesame Example: saccadic curvature Eye-tracker data → define DV’s in OpenSesame Example: saccadic curvature Pupillometry practical Pupillometry practical Assignment due: Sunday 1st 23:59 Pupillary light response: not just to our direct visual environment Also to memorized brightness (Mathôt et al., 2017: pupil response to semantic brightness of words) Pupillometry practical Assignment due: Sunday 1st 23:59 Pupillometry practical Assignment due: Sunday 1st 23:59 “have you seen this orientation?” H: pupil responds to brightness of memorized stimulus location Pupillometry practical Assignment due: Sunday 1st 23:59 From today’s module in Canvas, download pupillometry practical.pdf Assignment consists of 2 parts; Part 2 is about analyzing data (Thursday!) Linear mixed-effect models in R The logic of LMMs Explaining more variance than regular ANOVA’s The logic of LMMs Fixed effects vs. random effects Experimental variable (conditions) subjects items (stimuli) Those things about which we have hypotheses; (a specific direction of effect) Those things that we expect may be variable, but for which we do not expect a particular pattern Covariates for which we expect a particular pattern Both in terms of intercept and slope, i.e. overall performance and effect strength An example Cats, Dogs & Capybaras Fixed effect: picture-sound congruency (congruent vs. incongruent) Random effect: subjects, animal type, animal picture, sound (both in terms of intercepts as well as slopes) → The data will be analyzed on the basis of single trials, rather than averages per condition and participant Linear mixed models: The tutorial Place ‘workshop_data.txt’ in a folder of your preference Do the same for LMM_script.R A hypothetical study of the influence of alcohol on cognition - We test people’s ability to focus; how strongly are they distracted in a speeded reaction task? 40 participants are tested in 3 sessions each: one after drinking a litre of the finest beer, one after drinking a litre of malt, and one after drinking a litre of lemonade Per session they do 400 trials: 200 without being distracted, 200 with distractor We don’t want too many stimulus repetitions, so we have 200 different stimuli, each of which is being tested once per condition per participant. → A 2 x 3 design, with distraction (present vs. absent) and alcohol consumption (alcohol, placebo, control) as factors ...and a few other things that might be contributing variance Let’s take a look at our data R accepts various file formats This txt file is interpreted as a csv file. First row indicates column names (separated by comma) All subsequent rows are individual trials. And now: the script! To install LMM functionality: type install.packages(“lme4”) and press Enter To install LMM functionality: type install.packages(“lme4”) and press Enter To install LMM functionality: type install.packages(“lme4”) and press Enter To install LMM functionality: type install.packages(“lme4”) and press Enter R doesn’t always know whether your IV is a continuous variable or a factor (if condition names are numbers rather than names) specify that “distractor” is a factor The analysis is going to show the extent to which DV’s in one condition differ from another. You can choose your reference conditions And finally, the actual analysis... Printing the outcome (in the left window) Let’s run our script: Select all (cntrl + a or ‘apple’ + a) Then cntrl + r or ‘apple’ + r Let’s run our script: Select all (cntrl + a or ‘apple’ + a) Then cntrl + r or ‘apple’ + r b-value = 63.70 SE = 3.77 t = 16.91 t = | 1.96 | ≈ p = .05 Intercept can be left alone (it’s the ref. average, sign. different from zero) Let’s run our script: Random effects Select all (cntrl + a or ‘apple’ + a) A small bit of variance was explained by differences across stimuli (items); much more variance was explained by inter-subject differences. But the residual – i.e., variance not accounted for by these random effects, was even larger. Then cntrl + r or ‘apple’ + r Now you try: - Test for a main effect of session - Test for an effect of session on the error rate because error is a binomial variable, after “data=data,” we add family=“binomial” and instead of lmer(), we use glmer() Don’t forget to ‘inactivate’ the line of code that excludes incorrectly answered trials! (place a ‘#’ in front of it). - Test for the interaction between session and distractor on RTs. In the function, write session*distractor About interactions... The * operator tests interaction plus main effects, while the : operator tests only the interaction Be careful with main effects! That main effect of distractor is a main effect of distractor in the reference condition of session But now we want the perfect model! (1+distraction | subject) On the right of the vertical line: variable name On the left: 1 = random intercept (subjects may on average vary from one another) ‘distraction’ = random slope (the effect of distraction may vary across subjects) But now we want the perfect model! What do you think – do we want random slopes for subjects in the model? What about random slopes for items? But now we want the perfect model! There is no consensus about what is a perfect LMM My advice: let the structure of your model be theory-driven (Should we include day of the week? Nationality? Age?) It’s basically always reasonable to assume that effects may vary across subjects; so by default I’d include random slopes We can verify that a model with random slopes explains more variance than a model without random slopes Likelihood-ratio test model1 <- lmer(with random slope) model2 <- lmer(without random slope) anova(model1 , model2) The universe impacts our behavior in a virtually infinite number of ways Include the parts of the universe that you feel are really relevant How do you report your analyses? - “We ran a linear mixed-effect model (LMM) with A and B as fixed effects and by-C and by-D random intercepts as well as random slopes.” - “The maximal random structure that successfully converged was one that included C and D random intercepts and by-C random slopes for the distractor factor” How do you report your analyses? - “Values of t > | 1.96 | were deemed significant” How do you report your analyses? - Report b, SE and t. Mind positivity/negativity of b and t , as this indicates direction of effect - Example text will be shared on Canvas. Multi-dimensional cognition: Reading Practical points Pupillometry assignment due Wednesday 23:59 LMM assignment on Canvas tomorrow, due Wednesday next week 23:59 This week: finish experiment and start data collection Next week: collect data and write paper Today - Orthographic processing - Attention - Sentence processing - Instructions for report ~8000 BC: accountancy system ~3000 BC: logography & alphabet Why is reading interesting for the cognitive psychologist? Reading relies on many realms of cognition - Vision Attention Memory Language processing Oculomotor control How do these functions come together? How do these functions operate in applied contexts? One of the earliest topics of cognitive psychology 1886 Cattell 1908 Huey “[…] to completely analyze what we do when we read would almost be the acme of a psychologist’s achievements, for it would be to describe very many of the most intricate workings of the human mind.” What is language? Language vs. communication Communication: any transmittance of any signal in any perceptual modality Communication is the overarching thing; language is but a means to communicate What is language? Language is a hierarchical system Comprises building blocks that can be combined into building blocks that can be combined into building blocks… Comprises rules about how to combine building blocks at each level of the hierarchy… The set of structures that can be built following the rules is infinite Building blocks visual features > letters > words > sentences > context Does the brain have distinct processing stages for these various building blocks? Cognitive models: yes. Perception lecture: - various levels of processing - interactions among levels Top-down vs. bottom-up Orthographic processing The interface between letters and words Let’s go back to 1886… Letters in words are recognized faster than letters in nonwords PLUMP PMULP (word superiority effect) James McKeen Cattell And then a century forward… Words with large ‘orthographic neighborhoods’ are recognized faster How do we start processing a word? Orthographic processing ― Recognizing letters and their positions 1970’s: somewhere in the brain, we have an array of ‘slots’ Separate set of letter detectors for each slot. Letter detectors are activated by visual input How do we start processing a word? Orthographic processing ― Recognizing letters and their positions 1970’s: somewhere in the brain, we have an array of ‘slots’ Separate set of letter detectors for each slot. Letter detectors are activated by visual input Population receptive fields This back-and-forth process repeats until word detector reaches a recognition threshold But hold on a sec… Letters are flexibly encoded for their positions But hold on a sec… Letters are flexibly encoded for their positions! Potential solutions: - Positional noise: letters activate not only their slot but also surrounding slots - Bigram representations: an intermediate layer between letters and words, where (location-invariant) letter combinations are activated Positional noise versus bigrams Some recent research... How does the distance between two letters affect recognition of the bigram? abcdefgh abcdefgh C-D C-F Some recent research... How does the distance between two letters affect recognition of the bigram? abcdefgh abcdefgh C-D C-F “ Do you see the bigram CD ? ” “ Do you see the bigram CF ? ” Some recent research... How does the distance between two letters affect recognition of the bigram? abcdefgh abcdefgh C-D C-F “ Do you see the bigram CD ? ” Some recent research... How does the distance between two letters affect recognition of the bigram? abcdefgh abcdefgh C-D C-F “ Do you see the bigram CD ? ” “ Do you see the bigram CF ? ” “ Do you see the bigram DC ? ” “ Do you see the bigram FC ? ” : erroneous recognition of DC Some recent research... Data can only be explained by a combination of absolute position coding and bigrams! abcdefgh C-D abcdefgh C-F “ Do you see the bigram CD ? ” “ Do you see the bigram CF ? ” “ Do you see the bigram DC ? ” “ Do you see the bigram FC ? ” Some recent research... Population receptive fields aren’t small enough to know single letter locations A new theory of orthographic processing... PONG (the Positional Ordering of N-Grams ) - The brain is a sequence learner t, th, the, ther, there, here, ere, re, e - The brain estimates the laterality of N-grams through bi-hemispheric activation differences INTERMEZZO Report instructions Four sections: Introduction, Methods, Results, Discussion Separately: Abstract (brief ~200 word summary) that conveys the entire story Introduction: Question, background literature, hypotheses Methods: Participants, Expt. design, Apparatus, Procedure Results: Description of data cleaning procedures, analyses Discussion: Recap, Answer to question, critical evaluation INTERMEZZO Presentations in 2 weeks 5 mins per group – blitz talks Attention in reading Attention in reading When do we start processing a word? The key to a smooth read is to start processing a word already before looking at it Limits imposed by visual acuity: you’ll get some letters and visual cues such as word length (Rayner: preview benefit) Visuo-spatial attention (and acuity) Chfon du phin septonder a vory bock sphenctle woth polery send. Evel yoi e plofenint eetri snacks un prenk sciontofoc at efrtoi songle a you can read this quite well but hfon du phin septonder you cannot tell whether the text beyond evel yoi e plofenint entho is really meaningful at all bechefrtoi songle hfon du phin septonder a vory bock sphenctle woth polery send evel yoi e plofenint extri snacks un prenk a sivintisivin squer mitors. When do we start processing a word? Our visuo-spatial attention is not confined to the word at which we look directly. Covert attention moves ahead of the eyes to the next word. …or maybe attention was directed at multiple words from the get-go. A longstanding debate about attention in reading Rayner & co.: “Only one word is attended at a time” Joshua & others: “Maybe not” Serial processing of words? Flankers lexical decision task: Rayner: word recognition takes +-200 ms. …meaning there should be no time to process flankers Serial processing of words? We cannot prevent ourselves from processing the flankers! Only (sub-lexical) orthographic processing? Serial processing of words? Similar effects with semantically related flankers How to track covert attention (during reading)? Vary the brightness of flanker locations: Target recognition speed influenced by words on the left and right but not above and below… And pupil responds to brightness of words left & right, but not above & below the target! But the flanker paradigm is an artificial task. Maybe attention is distributed differently during normal text reading? Various papers: eye movements are unaffected by higher-order properties of upcoming words... so let’s look at brain activity Do readers process multiple words at once? The typical empirical strategy: This sentence is a simple example Do readers process multiple words at once? The typical empirical strategy: This sentence is a simple example Typical outcomes: - Word 1 influenced by letter overlap with Word 2 - Word 1 not influenced by frequency or semantics of Word 2 EEG: Fixation-related potentials Time-lock the electrophysiological window of interest to the start of a fixation on a target word EEG: Fixation-related potentials Time-lock the electrophysiological window of interest to the start of a fixation on a target word Prediction: Syntactic processing of the target word should be hampered by a syntactically incompatible adjacent word THE DOG JUMPED AWAY THE DOG YELLOW AWAY Methods → Any effect of the syntactic manipulation of word n+1 must have been triggered during the fixation on word n. Results: oculomotor data No effects in oculomotor data Results: oculomotor data No effects in oculomotor data Results: EEG data Readers process multiple words in parallel OB1-reader Sentence processing Syntax Additional evidence that we’re multiple words in parallel Syntax → Word superiority, and sentence superiority ‘man’ is recognized faster in the man can run than in run man the can But are we completely flexible? baby dog eats meat baby eats dog meat Our expectations constrain the mapping of words onto locations Unsolved questions... How does word position coding work? When does it happen? What factors influence it? baby dog eats meat baby eats dog meat Recap Our attention is directed to multiple words; not just to the word that we look at Letters and words are flexibly associated with locations Letters, words and sentences are separate things in the brain There is cross-talk between regions coding for letters, words and sentence structures