LC Developing Skills for Psychologists/Neuroscientists 1 PDF

University of Birmingham Dubai Campus LC Developing Skills for Psychologists/Neuroscientists 1 Week 09: Critical analysis 1.  Overview What is this thing called critical analysis? Using evidence to evaluate claims Numbers and chance (guess correcting MCQ exams) Probability and statistic...

University of Birmingham Dubai Campus LC Developing Skills for Psychologists/Neuroscientists 1 Week 09: Critical analysis 1.  Overview What is this thing called critical analysis? Using evidence to evaluate claims Numbers and chance (guess correcting MCQ exams) Probability and statistical evidence (p-values, sample sizes, effect sizes) Causal inference (research designs) What is this thing called critical analysis? What is this thing called critical analysis? At its simplest level, critical analysis is the evaluation of statements (Ennis, 1964) Individual differences in personality are completely determined by genetic inheritance “There are actually cities like Birmingham that are totally Muslim” (Fox News, 2015) Answering 25 questions correctly out of 50 on a 5-option MCQ exam means that the exam candidate knew the correct answer to half of the questions Ultra-processed food linked to 32 harmful effects to health, review finds (The Guardian, 2024) In Psychology and Neuroscience critical analysis almost always refers the strength of support for a statement of claim provided by evidence Numbers, probability and statistics Research designs (What do correlational observations tell us? What do experiments tell us?) Academic skills of critical evaluation are the central transferable skills which you gain on your degree programmes, applicable equally to the graduate workplace and everyday life. Using evidence to evaluate claims How does chance work? People often have trouble in thinking about chance.. (e.g., Rao & Hastie, 2023) Coin flips: flips are independent (no skill in outcome) and if the sides are evenly weighted, this should be 50% (p=0.5) for heads and 50% (p=0.5) for tails “Gambler’s fallacy” / “Hot hand fallacy”: If there have been lots of “heads” outcomes in a row, then you are more likely to get a “tails” outcome Usual Dependent sequences: chance isn’t involved to the weather same extent Gambler’s fallacy etc. may partly result from the Unusual assumption of dependence in sequences (as true weather randomness is fairly infrequent in the real world) MCQ Exams Imagine you are an examiner and you have set a multiple-choice question (MCQ) exam 50 questions, each with 2 options to choose a “single correct answer” from A candidate scores 25 out of 50 – should you assume that they knew the answer to half of the questions you set? What role might chance be playing here? Applying guess corrections (considering chance performance) In our example (50 single correct questions, 2 options each) If the candidate guesses at a 2-option questions they have a 50% chance (p=0.5) of selecting the correct answer If the candidate has no understanding at all of the module material, but they answer all questions, we should expect a score of 25 out of 50 by chance (50%, p=0.5) In the Developing Skills for Psychologists exam.. There are 50 single correct answer questions with 5 options each (why 5 instead of 2?) We apply a guess correction in which we take the maximum score (i.e., 50) and subtract the number of correct answers available by chance (i.e., 10), and this score is then scaled as a % of the new maximum score (40) Candidate 1: Gets 49 correct answers out of 50; 49 – 10 = 39; (39/40)*100 = 98% Candidate 2: Gets 39 correct answers out of 50; 39 – 10 = 29; (29/40)*100 = 73% Candidate 3: Gets 29 correct answers out of 50; 29 – 10 = 19; (19/40)*100 = 48% Candidate 4: Gets 19 correct answers out of 50; 19 – 10 = 9; (9/40)*100 = 22% Probability and statistical evidence Using probability as statistical evidence: Null Hypothesis Significance Testing  Sample size and effect size But NHST is not the only thing to be thinking about – we also care about whether what we have observed is significant in the real world. Sample sizes have an important bearing here: Small sample sizes will make null findings difficult to interpret (because an observation might have been significant if more participants were tested (i.e., with more statistical “power”) But as sample sizes grow, any given finding is much more likely to be statistically significant even if it is very small (and “insignificant” in the real world) Effect sizes are also important but often not reported (particularly in older work) (e.g., Cohen’s d / Partial Eta Squared) Sample size example… Small Sample Size: Large Sample Size Imagine you're testing a new drug to see if it lowers Now, let's say you conduct the same study with 1,000 blood pressure. You conduct a study with only 10 participants. participants. Result: You find a statistically significant difference in Result: You find no significant difference in blood blood pressure between the drug and placebo groups, but pressure between the drug and placebo groups. the difference is very small (e.g., a 0.5 mmHg reduction). Interpretation Issue: With only 10 participants, Real-World Significance: Even though the result is your study might lack the statistical power to statistically significant, the small reduction in blood detect a real effect. If you had more participants, pressure might not be meaningful in a clinical or practical you might have found a significant difference. sense. Can lead to null findings that are hard to interpret Increase the likelihood of finding statistically significant because the study might not have enough power to results, even for very small effects that might not be detect a true effect. important in the real world. Effect size Effect size is independent of sample size but provides an estimate of how substantial a difference or “effect” is in an observation. It is determined by the magnitude of a difference between numerical values (e.g., means) and the degree of variance (noise) in those observations Greater difference in means = larger effect size Smaller variation (noise) in means = larger effect size Causal inference (research design)  Exercise and Low cholesterol weight loss and mortality Does correlation mean causation? Does causation mean correlation? Watching violent TV and aggressive Drug use and psychiatric behaviour disorders  Observational studies vs. experiments Observational studies Experiments Excellent for observing associations in The go-to design for measuring causality directly naturalistic settings Experiments (conditions manipulated to measure Statistical tests usually assess correlational effects on outcomes) associations between variables (e.g., Quasi-experiments (e.g., age-group or gender personality and job satisfaction) comparisons, non-random allocation to Cannot measure causality directly conditions) establish somewhat weaker evidence But can be used to provide indirect indications of causation about causality with clever controls Involves heavy intervention so not necessarily Often require large sample sizes, especially to the best way to study what happens in the real explore potential causal associations world Can be impractical, often for ethical reasons  Experiments and causality: The randomised controlled trial approach Experimental Pre-test Post-test intervention Random Double blind intervention (representative) and assessment assignment Control Pre-test Post-test intervention Population Representative Intervention and sampling assessment Key features: Representative sampling, Random (representative assignment), Control condition/intervention, Double blind intervention, Double blind assessment, Comparison of outcomes in relation to pre-test Any questions?