Week 2 - Meta-Research PDF

Overview Study Participants WEIRD Populations Online vs. In-Person Populations Replication Crisis Replication Forecasting the Future Many Labs Project Why Replication Fails Study Participants Discussion Most studies are conducted on American college students who are enr...

Overview Study Participants WEIRD Populations Online vs. In-Person Populations Replication Crisis Replication Forecasting the Future Many Labs Project Why Replication Fails Study Participants Discussion Most studies are conducted on American college students who are enrolled in introductory-level psychology classes. How do these participants differ from other humans? Do these differences matter? WEIRD Populations Tend to come from societies that are: Western Educated Industrialized Rich Democratic Western Includes most of English-speaking world, and most of Western Europe Educated Most study participants are current college students, and most study designers have M.S. and/or Ph.D. Industrialized Most modern studies take place online, or require comfort working with computers Few effective ways to recruit participants without using the internet Rich Study participants more likely to have access to computers, cars, houses, etc. Democratic U.S., Australia, Canada, Western Europe: lots of representative governments where individuals expected to vote/protest/etc. Discussion Most studies are conducted on WEIRD populations. How do these participants differ from other humans? Do these differences matter? WEIRD Populations Compared to non-WEIRD populations… less cooperative more concerned with being unique less conforming greater risk-takers higher trait narcissism less collectivism focused on personal choice Online vs. In-Person Research Who here has participated in research? Online? In person? Online vs. In-Person Research Online Research Advantages Greater ability to recruit from different backgrounds than U.S. college students Online Research Disadvantages Inability to check that participants are paying attention or understanding the directions In-Person Research Advantages Ability to conduct high-intensity studies with lots of different types of variables In-Person Research Disadvantages Have to use convenience samples People the researcher can easily access, who volunteer in exchange for small rewards Replication Replication The ability for a study to be repeated, producing consistent results each time Shows a set of results is robust, and can be applied to real-world problems Replication Examples Greater exposure to advertising leads to less mental well-being among neurotypical participants Original study: Bauer, Wilkie, Kim, & Bodenhausen (2012) Replications: Forsell et al. (2019), Gilbert et al. (2020)  Replication Examples Active study strategies (self-testing, concept maps, summarizing) result in better test grades than passive re-reading Original study: Roediger & Karpicke (2006) Replications: McDaniel et al. (2007); Rowland (2014); Karpicke & Aue (2015) Replication Crisis  Bem (2011): Forecasting the Future Study claimed to find evidence of precognition — the ability to sense the future Extraordinary claims require extraordinary evidence Robinson (2011) tried to replicate the effect, found no evidence to support idea… Couldn’t get paper published Bem (2011): Forecasting the Future If a paper that manipulated numbers to (weakly) suggest ESP can get published… but a high-quality cautious re-do couldn’t… What does that say about the field? Many Labs Project (2014) Ran replications of 100 psychology studies 37 showed same results second time 28 showed weaker or murkier versions of original effect 35 did not replicate original effect at all Many Labs Project 2 (2018) Replicated 28 more psychology studies, based on media impact 14 showed same results second time 4 showed murkier versions of original effect 10 did not replicate original effect at all Why Replication Fails Why Replication Fails Roughly in order from most common reason to least common reason Measurement problems Science doesn’t deal in certainties Statistical errors Lack of information Publication bias Conflicts of interest Fraud Measurement Problems If you study aggression in mice, then try to apply this to aggression in humans Sometimes it will work (replicate) Sometimes it won’t Science doesn’t deal in certainties What does “p < 0.05” mean? Statistical Errors Researchers are humans, and humans make mistakes — especially with complex math Ex: Julia Strand and “mental squinting” Lack of Information Studies cannot be replicated if there isn’t enough information about them available 75% of authors who promised to share data refused on request >50% of authors with failed replications in Many Labs Project cited inability to reproduce methods Publication Bias Tendency for journals to publish only findings that support authors’ hypotheses… which distorts the entire research record Example 40 studies test whether ice cream cures depression 30 find no effect and go unpublished 10 find an effect and get accepted for publication 10 studies try to replicate the original effect 8 fail to replicate and go unpublished 2 replicate the effect and get accepted for publication Conflicts of Interest Authors who find support for shocking, novel, or financially lucrative hypotheses: Get more grant funding for future projects Will be offered jobs by more colleges Earn money from speaking engagements Creates incentives to not mention it if half your control group dropped out, or you found a weird coding error, etc. Conflicts of Interest Conflicts of Interest Fraud Hans Eysenck most famous psychologist ever to publish papers later found to be statistically impossible Peer review catches most, not all, cases