Evaluating Observational Studies (2023) PDF

Evaluating Observational Studies Abbey Krysiak, PharmD, BCPP  Explain the extent of biases in observational studies Define the need for critical appraisal of observational studies Objectives List the formal criteria to assess observat...

Evaluating Observational Studies Abbey Krysiak, PharmD, BCPP  Explain the extent of biases in observational studies Define the need for critical appraisal of observational studies Objectives List the formal criteria to assess observational studies Appraise design, methods, and analytical approaches in observational studies Evaluate the results of observational studies with randomized controlled trials Introduction Observational studies play an important role in medical literature Examine rare events and events that take time to develop Important role in comparative effectiveness research Provides a “realistic” indication of daily practice Introduction Critical review of observational studies: is important because of variable quality that threatens internal and external validity requires understanding of observational study design, their limitations and critical evaluation of the findings DO NOT blindly accept results! We need to know how to appropriately evaluate Outline for Today Brief review of the biases and confounding from observational study design Introduce formal criteria for critical assessment of observational study design Suggestions on how to compare results from observational studies with results from randomized clinical trials Bias and Confounding Bias and Confounding Bias are systematic errors in the way subjects are selected, measured and analyzed Selection bias Particularly problematic because treatment assignment is not randomized Error in the estimate of effect due to procedures used to SELECT subjects or factors that influence study participation or follow-up Occurs when subjects selected into the study are not representative of the population or the allocation of treatment is related to the outcome Measurement bias, also called information bias Occurs at data collection stage when there are systematic differences between study groups regarding how the exposures and outcomes are measured or reported by study participants, care givers, or researchers Bias and Confounding A confounding factor is associated with both the outcome and the treatment Is not a consequence of the treatment Selection bias can occur if confounding factors are not balanced between treatment and comparison groups Bias and Confounding Controlling for confounding factors: Observable confounding factors Stratification Statistical adjustment Unobservable confounding factors Instrumental variable methods Heckman selection model  Two broad issues should General Criteria for be considered: Evaluating Internal validity: Are the study results Observational valid? Studies External validity: Will the results generalize to local practice?  Strengthening the General Criteria for Reporting of Observational studies in Epidemiology (STROBE) Evaluating Statement that was developed that helps Observational evaluate observational studies Studies Helps readers critically appraise evidence  Checklist for Critical Assessment: General Criteria for Research question Study design Evaluating Study population Exposure and outcome Observational Data analysis and confounding Studies Presentation of results and interpretation Practice and policy implications Example This checklist was applied to the following observational study: Grodstein F, Stampfer MJ, Manson JE, et al. Postmenopausal estrogen and progestin use and the risk of cardiovascular disease. N Engl J Med. 1996;335(7):453-461 This prospective cohort design examined the effect of estrogen-progestin combination use on risk of cardiovascular disease (CVD) in postmenopausal women  Were the research questions, including any pre-specified hypotheses, clearly stated? Research Question Were the scientific background and rationale for investigating the research questions provided? Research Question Just like with other study questions: Must be plausible Should be noted in the introduction Should make clear the population being studied and the outcomes Scientific background and rationale should be discussed  Example: page 453 Authors state the research question within the title! Rationale found within the introduction Research 1. Previous evidence has shown that progestin use may increase risk of CVD Question: by increasing LDL cholesterol levels and lowering HDL levels, and reducing the beneficial effect of estrogen on arterial Application dilation and blood flow 2. There is insufficient information on the effect of estrogen-progestin combination use on CVD  Was the observational study design clearly stated? Study Design Was the choice of observational study design appropriate for the research question under investigation?  Cohort studies Case-control studies Study Four general observational study Design designs: Cross-sectional studies Consider if the design make sense with the stated research question Case-series studies or case reports Study Design Case-Control Study Cohort Study Study Design: Example Prospective cohort study design Application Goal was to infer causal relationship between hormone replacement therapy (HRT) and CVD Not clearly listed anywhere… Stronger design to examine causality in observational studies, so it is considered appropriate If outcome is rare or takes a long time to develop, you need a larger sample size and/or longer follow- up time Rarity of CVD was unknown in this population This study had a large number of patients (n=59,337) and a long follow up period (up to 16 years)  What was the source population and how does it compare to the target population underlying the research question? How were the participants selected and whether the inclusion and exclusion criteria were appropriate? Subject Have the characteristics of the study population been sufficiently described and how do these characteristics Selection compare to patients encountered at clinical practice? Was an appropriate comparison group used in the study? If yes, Was justification for selecting the particular comparison group provided? Is the comparison group appropriate to address the research questions under investigation? Is the comparison group comparable to the treatment group? Are there standing differences at baseline between the treatment and comparison groups that may lead to biased outcomes?  For primary data collection: Has the response rate been clearly reported and whether efforts were made to maximize participation rate? Have the investigators reported the characteristics between the respondents and non-respondents Subject Selection and whether any standing differences may lead to biased results? Has attrition rate over time been clearly reported and whether reasons for attrition and impact of attribution been discussed?  Crucial to understand how groups were selected in order to determine generalizability of results Understand the source from which the study population was selected May not represent patients in clinical practice Subject Selection Evaluate inclusion/exclusion criteria Stringent restrictions may help with reduce chance of confounding, but may limit generalizability  Comparison group should always be utilized Case-control: have “controls” that did not experience the outcome Cohort: also have comparison group to represent common changes over time in similar individuals Subject Selection Comparison groups should be matched to treatment groups Baseline characteristics should be reported to help assess the groups comparability and identify any confounding variables  Subject Selection Attrition rates Reasons for attrition should also be should be listed to determine if they are related reported to the outcome Should be utilizing Subjects analyzed in the group they an intention-to- are originally assigned treat analysis Subject Selection: Application Example: Subjects selected from the Nurses’ Health Study (NHS) Page 454 – description of NHS study Large cohort of nurses aged 30 – 55 years of age (n = 121,700) who were surveyed biennially since 1976 Study did not provide information on how nurses were selected… Nurses were selected to increase accuracy of self -reported medical information and long-term survey retention Results may not be generalizable because nurses may be more adherent to medications, may take preventative measures of CVD Subject Selection: Application Example: Page 454 - restricted the sample to women without pre-existing CVD or cancer on HRT This may help with possible confounding by pre-existing conditions BUT may muddy the results because those who initiated HRT use between surveys, but developed CVD before the survey would be excluded… why is this important? May underestimate the risk because those initiated on HRT between surveys but developing CVD prior to next survey would be excluded In clinical trials – CVD risk increases the first year of use, but decreases over time Baseline characteristics listed in Table 1, page 455 No statistical significant tests of differences reported  How are the exposure and outcome measured? Exposure Are the tools used to measure exposure and outcome accurate? and Outcome In cohort study, is the follow-up period sufficient to identify the outcome? Measures In case-control studies, is the look-back period appropriate to identify exposure?  Exposures usually measured using: Exposure Patient reports Medical records and Pharmacy claims data Outcome In case-control studies: Measures Important to consider the look-back period If an event takes a long time to develop (e.g. cancer), a short look-back period may underestimate exposure  Outcome measures Exposure Outcomes should be an objective measure should be appropriate for the study purpose and Sufficient details Outcome regarding measurement Measures process should be provided Exposure and Outcome Measures: Application Example: HRT exposure was ascertained through self-report by the participants… considered reliable Exposure measured as: Current estrogen-progestin combination use Current estrogen alone use Past use Never use Current dose of estrogen was further stratified at 4 levels to examine CVD risk by dose: 0.3mg, 0.625mg, 1.25mg, ≥1.25mg (table 4, page 458)  Nonfatal CVD events identified Exposure by participants’ and self-report Outcome Measures: Fatal CVD events reported by Application deceased participants’ family Data Analysis and Confounding Were statistical Were the statistical methods appropriate Were residual analysis appropriately for the study design confounding discussed adjusted for observed and measurement of and assessed? confounding? the outcome variables? Data Analysis and Confounding Appropriateness of statistical tests Sample size affects power of a statistical test Rare events → insufficient sample size = erroneous conclusions Confounding is a threat to internal validity Should be adjusted Multiple confounders → multivariable regression analysis Continuous data = linear regression model Binary data = logistic regression model Clinicians should also be aware of residual confounding bias resulting from unobservable confounders Data Analysis and Confounding: Application Example Sample size estimation was not done Both age-adjusted and multivariate-adjusted RR ratios were estimated Relative risk ratio (RR) was used as a measure of association Defined as the incidence rate of CVD among women in various categories of hormone use divided by the incidence rate among women who never used hormones Other potential confounding factors were explored Physician visits, socioeconomic factors Presentation of Results and Interpretation Are the interpretation and conclusion supported by the study findings?  Findings reported as: Risk ratio Odds ratio Hazard ratios Presentation of Results and Estimates of relative risk between treatment groups… may exaggerate the actual risk/benefit of Interpretation treatment Number needed to treat (NNT), number needed to harm (NNH) How Do I Report Findings?  Example: Relationship between HRT and risk of CVD presented as relative risk ratio 95% confidence intervals of relative Presentation of risks were reported Results and Investigators also reported total person years and number of cases for Interpretation: each RR calculation → can calculate absolute risk Application The age-adjusted RR of major coronary artery disease was estimated to be 0.45 between postmenopausal women who used estrogen alone and those who never used HRT, which would mean 55% reduction in risk HOWEVER the unadjusted rates were 0.6 and 1.4% - which would be a difference of 0.08% NNT = 1/0.0008 = 1,183  Application to What is the impact of the findings on Practice and Policy practice or policy?  Well designed observational studies can complement findings of RCTs Application Many black-box warnings issued by to Practice the FDA are based on evidence from observational studies and Policy Decisions on implications for practice should not be based on one studies findings  Example Authors concluded “the addition of progestin to estrogen does not appear to attenuate the Application to cardioprotective effects” At the time of this publication, no RCTs were available Practice and Findings were consistent with what was available at Policy: the time Choosing nurses may limit its generalizability Application There was a large relative risk reduction estimated, but the absolute risk was relatively small Overall conclusion: With a small absolute reduction and possible increased risk of breast cancer with the estrogen- progestin combination, caution should be taken with the use of HRT for primary prevention of heart disease  Generally consistent but occasionally may be Comparison of disconcordant. Results From Observational Studies and Randomized Trials Possible explanations for disconcordant results: Outcomes measured Study populations Exposure measured differently or differ differently detected at different rate Comparison of Results From Observational Studies and Randomized Trials Example Two RCTs → Heart and Estrogen/progestin Replacement Study (HERS) and Women’s Health Initiative (WHI) trials Found NO or INCREASED risk for both primary and secondary preventions Study population In HRT studies, differences in subjects’ age and time since menopause for initiation of HRT suggest explanation of discordant findings Exposure and outcome RCTs conducted under strictly controlled environments with close monitoring Observational studies may be affected by barriers to health care providers and/or patient’s decision to seek medical attention Questions?