Reading Papers Critically PDF

**[Reading papers critically]** **[Why do we read them critically?]** - Humans make mistakes, biases, untrustworthy etc. - We are trying to have a healthy skepticism around papers - Means we can avoid making our own mistakes **[General rules]** 1. **Be skeptical but understanding**...

**[Reading papers critically]** **[Why do we read them critically?]** - Humans make mistakes, biases, untrustworthy etc. - We are trying to have a healthy skepticism around papers - Means we can avoid making our own mistakes **[General rules]** 1. **Be skeptical but understanding** a. Failures arent usually deliberate fraud b. Usually due to bad training or incentives 2. **We judge the content. Not the presentation** c. If spelling etc isn't great, it doesn't mean the whole study is bad 3. **We need to look for replications** d. Looking for the effect being shown elsewhere 4. **The real info is usually in the methods and results** e. Abstracts, intros, conclusions give a simple answer, but not on how the study was conducted f. The limitation section can sometimes not be great for identifying the study limitation as they overlook the detailed ones, only going into broad ones Process of critical reading 1. [Assessing the question] - Is it clear/important that the hypothesis is supported and justified 2. [Assessing the design] - Is the approach to design the most appropriate? - We cant do experiments on everything 3. **[Assessing the validity]** - [Internal validity]: how certain can we be this is due to manipulating the independent variable - Experiments should have high internal validity due to randomization (placebo vs real) to eliminate extraneous variables (theoretically) - [Asses 'Risk of Bias'] - Might this be systematic error? - Example: catching the 5 fist rats and they are all the weakest of the group **[How to assess bias]** **SELECTION BIAS**: groups may differ at their base line which would have bias - Random allocation - Bigger sample sizes **PERFORMANCE BIAS**: groups treated differently in ways other than the variable of interest after allocation (hypothesis that group 1 will do better than group 2, if the researchers/ppts know this, they may affect performance due to SDB) - Blind allocation - Blind experimenters **DETECTION BIAS**: experimental outcomes are assessed differently between groups - Blind the analyzer of the data to stop over analyzing to their bias **REPORTING BIAS**: the significant results are more likely to be reported, rest are discarded - Pre-register studies (say ahead of time what you are going to do to hold yourself accountable) What if you cant tell if something is bias? Be wary in your own interpretation. Its better to report 4. **[Assessing study results]** - [Statistical analysis] - Mistakes occur in professional people - Look for over convoluted analysis -- looking to see if 1 is better than 2, then they start overanalyzing it (trying to prove what their bias is) - [Sample size] - 'law of small numbers' may lead to extreme events by chance - [Effect size] - Statistical vs meaningful significance - Statistical = number - Meaningful = is it actually helpful? - How precise is the estimate? Using standard errors, confidence intervals - P-[values] - Beware of values that are around 0.05, some will get as close as they can 5. **[Assess if the study is generalizable]** - How might the study sample affect the results - WEIRD studies (white, educated, industrialized, rich, democratic) - Psychology students, female 18-24 - Can these be applied to all? - Study setting raises similar questions: - Is a mock interview the same as a real life police interview? - Time periods - How applicable are findings in the tie=me line, does the events still affect the behaviour etc?