Lecture 5: Research Claims & Validity PDF

Research Claims & Validity  Evaluating research Type of Claim Frequency Association Causal Evaluate the Construct External Statistical Internal quality of the validity validity valid...

Research Claims & Validity  Evaluating research Type of Claim Frequency Association Causal Evaluate the Construct External Statistical Internal quality of the validity validity validity validity study Variables Constant  Variables Manipulated variable Measured variable  Variables Manipulated variable Measured variable  How do we define variables? Construct – conceptual variable of interest – E.g., relationship satisfaction, stress, mindfulness Operational definition – defines construct by specifying exactly how it is measured or manipulated – Relationship satisfaction as self-report – Stress as cortisol levels – Mindfulness as meditation How do we define variables?  Three types of research claims 1. Frequency claims 2. Association claims 3. Causal claims  Frequency claims Goal: describe the rate or frequency of a variable 10% of people in a 4 in 10 teens admit to committed relationship texting and driving met their partners through online dating  Association claims Goal: predict the relationship between two variables Relationship Gratitude satisfaction Gratitude is correlated with relationship satisfaction  Association claims Correlational study - a study that measures the relationship between two variables Relationship Gratitude satisfaction r = correlation coefficient; tells us the strength of the relationship between two variables Check out: https://rpsychologist.com/correlation/  Positive association r = 0.6, p < 0.05 Relationship satisfaction Gratitude  Negative association r = -0.6, p < 0.05 Relationship satisfaction Gratitude  No association r = 0.0, p = 0.98 Relationship satisfaction Gratitude Correlation does not equal causation! r = 0.53, p < 0.05  Association claims Gratitude is related to greater relationship satisfaction Relationship Gratitude satisfaction Predictor variable Outcome variable (measured variable) (measured variable)  Causal claims Goal: explain the relationship between variables E.g., Expressing gratitude increases relationship satisfaction Relationship Gratitude satisfaction Independent variable Dependent variable (manipulated variable) (measured variable)  Causal claims 6 5 Relationship Satisfaction 4 3 2 Gratitude is manipulated by the 1 researcher 0 No gratitude Gratitude  Causal claims Criteria for making a causal claim: 1. Covariance - cause and effect co-occur 2. Temporal precedence - cause precedes effect 3. Internal validity - rules out confounds/ third variables Causal claims can only be made with experiments. Identifying Association Claims vs. Causal Claims  Research Goals & Claims Research Goal Research Claim Describe Frequency claim Predict Association claim Explain Causal claim  Research Goals & Claims Describe: how common are selfies? – How many people take selfies? – Claim: frequency Predict: relationship between narcissism and selfies – Do people higher in narcissism take more selfies? – Claim: association Explain: why are selfies and narcissism linked? – Does taking more selfies make people more narcissistic? – Claim: causal  How can we evaluate research? 1. Construct validity 2. External validity 3. Statistical validity 4. Internal validity  Construct validity How well a conceptual variable is operationalized i.e., Are you measuring what you want to be measuring? stress  External validity Are the findings generalizable to other people, contexts, and methods than those in the original study?  Statistical validity The extent to which a study’s statistical conclusions are accurate and reasonable.  Statistical validity Questions to ask Where to look Is the finding statistically significant? p-value Is the effect meaningful? Effect size Was the sample size large enough? Sample size (N) Power  Statistical validity Statistical validity of a frequency claim Margin of error (confidence interval) E.g., The CDC reports that 41% of teens text while driving. The margin of error is +/–2.6 %. Lower 41% Upper limit limit 38.4% 43.6%  Statistical validity Statistical validity of association and causal claims What is the strength of the association? → Effect size  Statistical validity Statistical validity of association and causal claims How precise is the estimated association? → Confidence intervals E.g., Gratitude is positively correlated with relationship satisfaction, r = 0.30, 95% CI [0.26, 0.34], p <.05  Statistical validity Errors in decision making Type I error: False positive – Mistakenly conclude that there IS an association when there actually is no association in the population – Possible whenever we reject the null (p <.05) Type II error: False negative – Mistakenly conclude there is NO association when there actually is an association in the population – Possible whenever we retain the null (p >.05)  Statistical validity Importance of a large sample size (N ) – Smaller margin of error/ confidence intervals (more precision) – More power to detect an effect – Less chance of Type II error  Internal validity Did you eliminate third-variables, or alternative explanations, for your findings?  Internal validity Three criteria for Establishing Causation Between Variables A and B 1. Covariance: as A changes, B changes 2. Temporal precedence: the manipulated variable (A) comes before the measured variable (B) in time 3. Internal Validity (also called control for confounds): there’s no other explanation for the change in B than A  Internal validity Independent variable: the causal variable; what we think affects the outcome In experiments, this is the variable that the researcher manipulates Dependent variable: the outcome of interest; this is the effect; this variable depends on the independent variable  Internal validity Random assignment: Every person in the sample has an equal chance of being selected for each condition  Tradeoffs Internal validity External validity  Evaluating the different types of claims Type Frequency Association Causal Construct How well the How well the variables How well the independent variable is are measured variables is manipulated and measured how well the dependent variable is measured External Generalize: people, Generalize: people, Generalize: people, contexts, contexts, methods contexts, methods methods Statistical Margin of error Effect size; strength of Effect size; the size of the association difference between groups Internal N/A N/A Temporal precedence Control of confounds  Evaluating Research Practice Theory: Violent videogames increase aggression Hypothesis: Participants play a violent videogame will be more aggressive after than participants who play a non-violent game. Study N = 200 Random assignment to violent videogame vs. solitaire After, chance to aggress with hot sauce Results: d =.20, p =.01  Evaluating Research Practice Construct validity: are you measuring what you think you are measuring? = aggression?  Evaluating Research Practice External validity: are your findings generalizable to other people, contexts, methods, etc.?  Evaluating Research Practice Statistical validity: Is the finding significant? Is your effect size large? Was the sample size large enough? Statistical significance: p =.01 Sample size: N = 200 Effect size: d =.20  How can we evaluate research? Internal validity: Did you eliminate ‘third- variables’ or other explanations for your findings? – Example: did your study introduce both violence AND competition? Violence AND excitement?  Evaluating research Type of Claim Frequency Association Causal Evaluate the Construct External Statistical Internal quality of the validity validity validity validity study Practice questions