Epidemiology Study Designs

Cross-Sectional Studies

• Compare current disease status and current exposure
• Strengths: Can study rare outcomes, faster, and no recall bias
• Weaknesses: Assignment to comparison group is not random, no assessment of temporality, and only assessing one outcome

Observational Studies

• Can study any question, less expensive or faster than intervention studies
• Limitations: Not randomly assigned to exposure groups, investigate correlation, not necessarily causation

Appraising the Quality of Observational Studies

• Recruitment: Do participants reflect the population of interest? (Selection Bias)
• Assessment of exposure: accurate, subjective or objective, validated, and measurement or classification bias
• Consideration of confounding factors

Systematic Reviews

• Strengths: Explicit and rigorous methods, scientific investigation with pre-planned methodology, enormous effort to minimize bias, capture the big picture of evidence on a topic, and meta-analysis allows for a larger sample size
• Limitations: Only as good as the available studies, publication bias, lack of research on a topic, and can't replace good clinical reasoning

What Makes a Strong Systematic Review?

• Clear question
• Comprehensive search
• Assessment of study quality

N of 1 Study

• Strengths: Look at real-world use of an intervention, allows for individualization, and compare naturopathic and conventional treatments
• Limitations: Doesn't work if the condition is curable or self-limiting, findings may not be generalizable, ethics, and high cost

Preclinical Studies

• Strengths: Allow for creativity and innovation, study possible adverse events or interactions, and high level of control
• Limitations: May not be clinically applicable to humans, highly controlled, and one isolated part of the story

Considerations

• Is pre-clinical evidence sufficient to guide clinical recommendations?
• Assessing risk of bias in individual studies and the body of evidence as a whole

GRADE Framework

• Imprecision: Are the results due to chance? Focus on 95% confidence interval
• Inconsistency: Many similar studies showing a consistent effect increase certainty
• Indirectness: Certainty is downrated when the intervention of interest is not studied in the population of interest and reporting the outcome of interest
• Publication Bias: Is this really all of the research evidence that exists?

What Makes Evidence More Certain?

• Large magnitude of effect
• Dose-response gradient
• All residual confounding would increase our confidence in an effect
• Ex. A very large observational or non-randomized study without other limitations

Compliance

• Should be assessed using simple methods like diary, pill count, phone calls, and questioning, or biological methods like blood/urine levels (although costly)

Withdrawals

• Important to account for missing data, with two approaches:
  • Per-protocol analysis: only analyzes people in the intervention group who completed the study
  • Intention to treat analysis: analyzes everyone in the group they were randomized to, even if they didn't complete or ended up in the other group

Risk of Per-Protocol Analysis

• Can lead to biased results due to systematic factors impacting who drops out (e.g., side effects, acceptability, impact on outcome)

Intention to Treat Analysis

• Techniques: last observation carried forward, statistical approaches
• Best way to minimize bias from dropouts
• Cautious, under-estimates effect (vs. per-protocol analysis, which over-estimates effect)

RCT Limitations

• Difficult to recall bias
• Only assessing one outcome
• Participants may not be representative of the population

Cross-Sectional Studies

• Compare current disease status and current exposure
• Strengths: can study rare outcomes, faster, no recall bias
• Weaknesses: assignment to comparison group is not random, no assessment of temporality, only assessing one outcome

Observational Studies

• Strengths: can study any question, less expensive or faster than intervention studies
• Limitations: not randomly assigned to exposure groups, investigate correlation, not causation

Appraising Quality of Observational Studies

• Recruitment: do participants reflect the population of interest? (selection bias)
• Assessment of exposure: accurate? (measurement or classification bias)
• Consideration of confounding factors?

Systematic Reviews

• Explicit and rigorous methods to identify, critically appraise, and synthesize evidence
• Informed consent, consider alternatives, monitor patient response and safety, alter treatment if needed

Grading the Evidence

• GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) framework
• Transparent framework for developing and presenting summaries of evidence
• Systematic approach to clinical decision making

How GRADE Works

• Clinical question (PICO format)
• Systematic review provides an estimate of the effect size of an outcome
• Author rates the quality of the evidence and strength of recommendations

GRADE Certainty Rating

• Certainty: whether an estimate of association or effect is correct or true
• Five levels of certainty: very low, low, moderate, high, and very high

What Makes Evidence Less Certain?

• Risk of bias
• Imprecision
• Inconsistency
• Indirectness
• Publication bias

What Makes Evidence More Certain?

• Large magnitude of effect
• Dose-response gradient
• All residual confounding would increase confidence in an effect
• Large observational or non-randomized studies without other limitations

GRADE: 2 Parts

• Certainty of Evidence
• Recommendation Strength
  • Consider evidence, benefits/harms, equity, resources, feasibility, acceptability

Selection Bias

• Sample confounding factors: age, gender, family history, comorbidities, BMI, socioeconomic status, marital status, education, exercise, diet, and other known or unknown factors
• Blinding: removes expectation bias; types of blinding include single blinding (participants), double blinding (person delivering the intervention), and triple blinding (person assessing the outcome)
• Clearly defined population: participants are a sample of a larger population; inclusion and exclusion criteria help define the population; consider generalizability (e.g., more or less ill, different ethnicity or geographic location)

Bias in Measurement of Outcomes

• Reporting of results: includes power calculation, reporting of planned measurements, and consideration of missing data and p-values
• Systematic review: a scientific investigation with pre-planned methodology to minimize bias; allows for the creation of a larger sample size and captures the big picture of evidence on a topic
• Limitations of systematic review: only as good as the available studies; can't replace good clinical reasoning

N of 1 Study

• Strengths: looks at real-world use of an intervention; allows for individualization and comparison of naturopathic and conventional treatments
• Limitations: doesn't work if the condition is curable or self-limiting; findings may not be generalizable; high cost and ethical concerns

Preclinical Study

• Strengths: allows for creativity and innovation; investigates mechanism of action; studies possible adverse events or interactions
• Limitations: may not be clinically applicable to humans; highly controlled; only one isolated part of the story

Grading the Evidence

• GRADE: a transparent framework for developing and presenting summaries of evidence; systematic approach to clinical decision making
• Grading of evidence: considers certainty of effect, imprecision, inconsistency, indirectness, and publication bias
• Recommendation strength: considers evidence, benefits/harms, equity, resources, feasibility, and acceptability

Certainty of Evidence

• Risk of bias: bias in individual studies; many tools for assessing risk of bias
• Imprecision: are the results due to chance? Focuses on 95% confidence interval
• Inconsistency: compares results of individual studies; looks at forest plots and statistical heterogeneity
• Indirectness: considers if the intervention was studied in the population of interest and reported the outcome of interest
• Publication bias: considers if all relevant studies were included; uses visual and statistical methods for assessment

What Makes Evidence More Certain?

• Large magnitude of effect
• Dose-response gradient
• All residual confounding would increase confidence in an effect
• Ex. A very large observational or non-randomized study without other limitations

GRADE: 2 Parts

• Certainty of evidence: how likely is it that something works?
• Recommendation strength: should it be recommended for use? Consider evidence, benefits/harms, equity, resources, feasibility, and acceptability

Statistics

• Define terms: sample, population, incidence, prevalence, and power

Population vs Sample

• Population: entire group of interest
• Sample: subset of population for study

Prevalence and Incidence

• Prevalence: total number of cases suffering from a disorder (e.g. 26,630 HIV-positive persons in Ontario in 2008)
• Incidence: number of new cases suffering from a disorder (e.g. 1,620 new HIV infections in Ontario in 2008, ~0.0001 or 1 in 10,000)

Hypothesis Testing

• Process of deciding whether study findings reflect chance or real effect
• Involves:
  • Difference in group means (averages)
  • Difference in likelihood of an outcome
  • Statistical significance (real differences or chance?)

Statistical Significance: P-Value

• P-value: probability of difference being due to chance
• Generally, 0.05 is the cut-off (5% likelihood)
• If p > 0.05, not significant; if p < 0.05, significant

Statistical Significance: CI

• 95% Confidence Interval (CI): range of values within which the true result is likely to lie
• If CI intervals overlap, non-significant difference; if they don't, significantly different

Relative Risk

• Probability of a bad outcome in the intervention group divided by probability of a bad outcome in the control group
• RR = 1: no difference; RR > 1: increased risk; RR < 1: decreased risk

Relative Risk Reduction

• How much did the treatment reduce the risk of the bad outcome?
• RRR = 1 - relative risk

Absolute vs Relative Risk Reduction

• Absolute RR: absolute amount the intervention decreased the risk
• ARR = risk of outcome with placebo - risk with mediation
• ARR and RRR can be misleading when outcomes occur infrequently

Odds Ratio

• Measures the relationship between an exposure and an outcome
• Often used in observational studies
• OR = 1: exposure does not affect the likelihood of the outcome; OR > 1: exposure associated with increased likelihood of the outcome

Outcomes

• Clinical outcomes: actual events or conditions (e.g. stroke, suicide, hip fracture)
• Surrogate outcomes: indicators of disease progression, not clinical events (e.g. biomarkers, LDL-Cholesterol, Columbia Suicide Severity Scale)
• Surrogate outcomes must be validated to ensure they predict the true outcome

Clinical Significance

• Does the intervention have a genuine, measurable effect?
• Related to extent of change and whether it makes a real difference in patients' lives
• No broadly accepted standard for clinical significance

Power

• Ability of a study to detect a real difference between two groups
• Depends on sample size, ability to detect differences, and extent of change