HTH SCI 3C04 Research Designs & Methods Winter 2024 PDF

HTH SCI 3C04 Research Appraisal & Utilization in Evidence Informed Decision Making Week 2 Week 2 Quantitative Research Designs, Measurement & Bias 2.1 Important Background Assessing Causation Hills Criteria of Causation (Hill, 1965) • Hill was a British medical statistician (1897-1991) who dev...

HTH SCI 3C04 Research Appraisal & Utilization in Evidence Informed Decision Making Week 2 Week 2 Quantitative Research Designs, Measurement & Bias 2.1 Important Background Assessing Causation Hills Criteria of Causation (Hill, 1965) • Hill was a British medical statistician (1897-1991) who developed 9 criteria for determining the causal link between a specific factor and a disease • Uses of Hill’s criteria: ➢ basis of epidemiological research, which attempts to establish scientifically-valid causal links between potential disease agents and many diseases ➢ identification of study designs offering strongest evidence ➢ appraising evidence obtained from multiple studies • Keep these criteria in mind as you consider the quantitative study designs & their relative strengths & weaknesses Hill’s Criteria for Assessing Causation Nine (9) Criteria: 1. 2. 3. 4. 5. 6. 7. 8. 9. Temporal Relationship Strength of Association Dose-Response Relationship Consistency of Association Biological Plausibility Experimental Evidence Alternate Explanations Specificity Coherence Hill’s Criteria (Causation) 1. Temporal Relationship (Temporality) • Exposure always precedes the outcome. • For example, if factor “A” is believed to cause a disease, then factor “A” must always precede the occurrence of the disease. 2. Strength of Association • The stronger the association, the more likely that the relationship between “a” and “b” is causal. • For examples: ➢ ➢ the higher the correlation between sodium consumption and hypertension, the stronger the relationship between sodium and hypertension. the higher the odds of 30-day hospital re-admission in post-surgical patients, the stronger the relationship between surgery and 30-day hospital re-admission. Hill’s Criteria (Causation) 3. Dose-Response Relationship • An increasing amount of exposure (dose), increases the amount of response (outcome). • If a dose-response relationship exists  stronger evidence for causal relationship (compared to absence of relationship). • Similarly, if a certain factor is the cause of a disease, the incidence of a disease should decline when exposure to the factor is reduced or eliminated. Hill’s Criteria (Causation) 4. Consistency of Association • Finding a consistent relationship between a factor and outcome across different studies with different populations  stronger evidence of causal relationship (compared to inconsistent findings) 5. Biological Plausibility • Association agrees with accepted understanding of pathological processes (i.e., there needs to be a theoretical basis for the association) Hill’s Criteria (Causation) 6. Experimental Evidence • The association between exposure and outcome can be supported with experimental evidence 7. Alternate Explanations • Determine the extent to which researchers have taken other possible explanations into account and ruled out these alternate explanations. Hill’s Criteria (Causation) 8. Specificity • • • Established when a single cause produces a specific effect. Weakest of criteria for causation Absence of specificity does not negate a causal relationship 9. Coherence • The association should be compatible with existing theory and knowledge. 2.2 – Features, Advantages & Disadvantages of Different Quantitative Designs Study Design A Study Design: • refers to the way a study is organized/constructed & methods used Quantitative study designs are best for questions about: • • • • • • cause of a disease (etiology) prognosis diagnosis prevention treatment economics of a health problem Quantitative Study Designs: Another Hierarchy Most Rigorous (least biased) Experimental Designs Quasi-experimental Designs Non-experimental Designs Randomized Controlled Trial: Continuous Outcome Eligible patients Exposure Outcome Experimental Intervention Outcome Control Intervention Outcome Randomization • Participants randomly allocated to control & treatment (intervention) groups • Participants followed forward in time (prospectively) from exposure to outcome Randomized Controlled Trial: Discrete Outcome Exposure Outcome Outcome Experimental Intervention Eligible patients Randomization Control Intervention No outcome Outcome No outcome • Participants randomly allocated to control & treatment (intervention) groups • Participants followed forward in time (prospectively) from exposure to outcome What are the strengths & weaknesses of the RCT design? RCT Design Strengths & Weaknesses STRENGTHS • Random allocation: to ensure groups are similar (known and unknown factors) (Hill’s Criteria #7 – Alternate Explanations) WEAKNESSES • Cost • Temporality1: longitudinal follow-up to ensure • Length of time required for follow-up until that exposure precedes outcome (Hill’s outcome observed Criteria #1 - Temporality) • Generalizability: patients that agree to participate may differ from the target population to whom the study is intended to apply • Ethics & Feasibly: Not always ethical or logistically feasible/suitable 1 temporality is one of the criteria for establishing causality – i.e., exposure occurs before outcome Cohort Analytic Study: Continuous Outcome Eligible patients Non-random allocation; choice or happenstance Exposure Outcome Exposed: Experimental Intervention Outcome Not Exposed: Control Intervention Outcome • Unlike in RCTs, participants are not randomly allocated to control & treatment (intervention) groups • As in RCTs, participants are followed forward in time (prospectively) from exposure to outcome Why is Randomization not Always Possible? • Ethical issues: • Is it fair to deny the intervention to the control group? ➢ Example: denying any cancer patient access to a drug that offers hope with no known risks (or minor risks relative to prognosis without the drug)? • Feasibility issues: • Is it feasible to deliver the intervention to only some members of the eligible population? ➢ Example: studying the effects of a new management system for hospital nursing staff (usually this type of intervention is implemented hospital-wide, so you would need to compare your hospital with the intervention to another one that has not implemented the intervention yet) What are the strengths & weaknesses of the Cohort Analytic design? Cohort Analytic Design: Strengths & Weaknesses STRENGTHS • Cost/Ethics & Feasibility: potentially less costly and/or more feasible than an RCT (if not ethical/feasible to randomize) WEAKNESSES • Selection bias: groups may differ in ways other than exposure to the intervention (Hill’s Criteria #7 – Alternate Explanations) • Temporality1: longitudinal follow-up to ensure • Baseline differences: groups may differ in that exposure precedes outcome (Hill’s characteristics that existed before the Criteria #1 – Temporality) intervention began (Hill’s Criteria #7 – Alternate Explanations) • Cost: Remains relatively expensive 1 temporality is one of the criteria for establishing causality – i.e., exposure occurs before outcome Cohort Design: Continuous Outcome Exposure Outcome Eligible patients with exposure Outcome • Interested in the likelihood that people will experience or develop an outcome given their exposure to a disease, condition or situation (prognosis question) ➢ e.g., how likely are patients with ulcerative colitis to develop bowel cancer? • Comparator? What are the strengths & weaknesses of the Cohort Design? Cohort Design Strengths & Weaknesses STRENGTHS WEAKNESSES • Temporality1: longitudinal follow-up to • Expense: follow-up may require large numbers of people ensure that exposure precedes outcome to see outcome (Hill’s Criteria #1 –Temporality) • Rare or specific exposures • Lack of a control group: (participants may systematically differ on predictors of outcome) (Hill’s Criteria #7 – Alternate Explanations) • Multiple outcomes • Inefficient: for studying rare outcomes • Best evidence for evaluating risk/prognostic factors • Time: follow-up may take a long time for diseases with a long latency period • Limited exposures: Can study only single or specific groups of exposures • Contamination: long follow-ups mean other factors may change that could cause the outcome (e.g., differential 1 temporality is one of the criteria for establishing carecausality & treatment, exposure to other causative – i.e., exposure occurs before outcome agents) (Hill’s Criteria #7 – Alternate Explanations) Case-Control Study Outcome Exposure Outcome (cases) Exposure No Outcome (controls) Exposure Eligible patients • Participants with and without the outcome are identified • Investigators look back in time (retrospectively) from outcome to exposure • Investigators often try to match cases & controls: this helps to ensure groups are as similar as possible regarding important variables that may influence the outcome (e.g., age, sex) What are the strengths & weaknesses of the Case-Control Design? Case-Control Design: Strengths & Weaknesses STRENGTHS WEAKNESSES • Cost & Time: often more economical and quicker than a cohort study • Uncertainty about Temporality1: difficult to confirm temporality (Hill’s Criteria #1 – Temporality) • Good for study of rare outcomes and common exposures • Exposure information: difficult to obtain accurate information on the timing, duration and dose of exposure (recall bias) (Hill’s Criteria #2 & #3 – Strength of Association, Dose-Response Relationship) • Includes control group (limits bias) (Hill’s Criteria #7 – Alternate Explanations) • Control group: difficult to find control group that is comparable (on factors other than the intervention) (Hill’s Criteria #7 – Alternate Explanations) • 1 temporality Can examine multiple exposures (e.g., if causality • Cannot examine multiple outcomes is one of the criteria for establishing – i.e., exposure occurs before outcome population at risk is not known) Descriptive/Cross Sectional Survey • A group of people are interviewed or asked to complete a survey to determine whether they have experienced an exposure of interest and an outcome of interest • Exposure & outcome are measured simultaneously ➢ e.g., group of women interviewed to determine their use of video display terminals & whether they had a miscarriage What are the strengths & weaknesses of the Cross-sectional/ Observational Design? Cross-sectional/Observational Design: Strengths & Weaknesses STRENGTHS WEAKNESSES • Cost & Time: economic and quick – don’t follow people over time • No Temporality1: outcome and exposure measured at same time (Hill’s Criteria #1 – Temporality) • Good for the exploratory research phase: can help to identify potential causal factors that can be studied with stronger designs • Exposure information: difficult to obtain accurate information on the timing, duration and dose of exposure (recall bias) (Hill’s Criteria #2 & #3 – Strength of Association, Dose-Response Relationship) • Control group: there may be no control group, or if there is it may not be comparable (Hill’s Criteria #7 – Alternate Explanations) 1 temporality is one of the criteria for establishing causality – i.e., exposure occurs before outcome Questions & Designs Typically, questions about: The effectiveness of prevention and treatment interventions: • RCTs are the most rigorous (strongest) design The cause of a health problem (causation): • RCTs are the most rigorous design • IF random allocation is unethical THEN cohort analytic design (non-random allocation) is used • IF an outcome is rare or takes too long to develop THEN case-control design is used • The course of a health state or condition (prognosis): • A cohort study is used 2.3 – Sources of Bias in Quantitative Research Studies Common Types of Bias PUBLICATION: SELECTION: EXPOSURE & RECALL BIAS: MEASUREMENT: CONFOUNDERS: • Bias against negative findings (not published) • Related to sampling • Those who get into either group should not be different from each other or the population they represent • Studies can struggle to confirm temporality of exposure and/or exposure characteristics (e.g., case-control studies) • Instruments: should be validated & reliable for study population • Interviewer: interviewers aware of the status of participants may probe more or less deeply • An alternative explanation of the findings • occurs when a factor is related to BOTH the exposure and the outcome & is not recognized and/or controlled • can make effects seem larger (or smaller) than they really are • Example: strong relationship between coffee consumption & lung cancer is confounded by smoking (a likely cause of both) Ways to Minimize Bias • Search for unpublished results (e.g., find abstracts of conferences, call researcher) • Blinding/masking (participants, interventionists, data collectors, outcome assessors, data analysts) • Strict adherence to research protocol: • establish and publish the study protocol • ensure rigorous study design • develop prospective hypotheses and analytical plans • applies to intervention procedures and measuring outcomes • Strict follow-up of participants; compare groups; compare completers to dropouts • Careful matching/control groups 2.4 Important Key Words 2.4.1 Dependent vs. Independent Variables Dependent vs. Independent Variables • An independent variable is the presumed cause, whereas the dependent variable is the presumed effect. • In an experiment: ➢ the independent variable is the variable that is varied or manipulated by the researcher, and the dependent variable is the response that is measured. • In non-experimental research (where there is no experimental manipulation): ➢ the independent variable is the variable that 'logically' has some effect on a dependent variable. For example, in research on cigarette-smoking and lung cancer, cigarette-smoking is the independent variable. Example #1: Select the Independent and Dependent Variable in the Scenario "There will be a statistically significant difference in graduation rates of atrisk high-school seniors who participate in an intensive study program as opposed to at-risk high-school seniors who do not participate in the intensive study program.“2 • Independent variable(s)? • Dependent variable(s)? 2LaFountain & Bartos, 2002, p. 57 Example #1: Select the Independent and Dependent Variable in the Scenario "There will be a statistically significant difference in graduation rates of atrisk high-school seniors who participate in an intensive study program as opposed to at-risk high-school seniors who do not participate in the intensive study program.“ • Independent variable: Participation in study program (Y/N) • Dependent variable(s): Graduate (Y/N) Example #2: Select the Independent and Dependent Variable in the Scenario "A director of residential living on a large university campus is concerned about the large turnover rate in resident assistants. In recent years many resident assistants have left their positions before completing even 1 year in their assignments. The director wants to identify the factors that predict commitment as a resident assistant (defined as continuing in the position a minimum of 2 years). The director decides to assess knowledge of the position, attitude toward residential policies, and ability to handle conflicts as predictors for commitment to the position.“3 • Independent variable(s)? • Dependent variable(s)? 3 LaFountain & Bartos, 2002, p. 8 Example #2: Select the Independent and Dependent Variable in the Scenario "A director of residential living on a large university campus is concerned about the large turnover rate in resident assistants. In recent years many resident assistants have left their positions before completing even 1 year in their assignments. The director wants to identify the factors that predict commitment as a resident assistant (defined as continuing in the position a minimum of 2 years). The director decides to assess knowledge of the position, attitude toward residential policies, and ability to handle conflicts as predictors for commitment to the position." • Independent variable: Knowledge of Position, Attitude Towards Policies, Ability to Handle Conflict • Dependent variable: Commitment to Position for 2+ years (Y/N) 2.4 Important Key Words 2.4.2 Data Types (Levels of Measurement) Four Levels of Measurement …yes, another hierarchy Ratio Interval Ordinal Nominal Attributes are only named; weakest Attributes can be ordered Distance is meaningful Absolute zero Why are Levels of Measurement Important? • Helps interpret the data: if a variable is nominal (categorical), then you know that the numerical values are just codes for the names. • Helps determine correct statistical analyses: ➢ if a variable is nominal, you would not report means but instead report the number & proportion of sample having each value and could use the chisquare test to examine group differences ➢ If a variable is ratio, you would report means and you could use a t-test to examine group differences ➢ different types of regression methods apply to different types of outcome variables 2.4 Important Key Words 2.4.3 Reliability & Validity Assessing the Quality of a Measure Reliability (stability, repeatability, precision) • The degree to which the measure is consistent, or gives the same result on two or more occasions (also considered stability) Validity (accuracy) • The degree to which a measure measures what it is intended to measure Reliability & Validity The bull’s eye is the TRUE value that you are trying to measure Relationship between Reliability & Validity • A measurement must be reliable to be valid. • Reliability is not a guarantee of validity. • For example, if someone could measure your height and say they are measuring intelligence, they would get a consistent and reliable result each time but this would NOT be valid since height is not an indicator of intelligence. Why are Reliability & Validity Important? If a study lacks reliability: • results will vary in further replications of the study • there is high level of uncertainty in answering the research question • uncertain results cannot be used to generalize If a study lacks validity: • it does not measure what we want it to measure, so the results cannot be used to answer the research question • the results also cannot be used to generalize 2.5 – Ethics in Choosing a Study Design Foundational Sourcesfor for Ethics Ethics Principles Foundational Sources Principles in Medical Research in Medical Research • International Codes of Ethics: ➢ Nuremberg Code (1947) ➢ Declaration of Helsinki (1964, Updated in 2013) ➢ Belmont Report (1979) • Country-Specific Ethics Policies: ➢ Tri-Council Policy Statement 2 (TCPS2) (Canada) ➢ Common Rule (United States) TCPS 4 2 (Canada) Respect for human dignity through 3 core principles: 1. Respect for Persons ➢ Participants autonomy (free, informed, ongoing consent) ➢ Protecting vulnerable people (incapable of autonomy) 2. Concern for Welfare ➢ Impacts to health or circumstances (physical, economic, social) 3. Justice ➢ Treating people fairly, equitably ➢ Recruitment process important 4 Tri-Council Policy Statement 2 (TCPS 2) (2018), Updated 2020 References Hill, Austin Bradford (1965). “The Environment and Disease: Association or Causation?,” Proceedings of the Royal Society of Medicine, 58, 295-300. LaFountain, R. M., & Bartos, R. B. (2002). Research and statistics made meaningful in counseling and student affairs. Brooks/Cole. Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2 - 2018). Date Accessed: December 14 2020. Website: https://ethics.gc.ca/eng/policy-politique_tcps2-eptc2_2018.html