Evaluating Systematic Reviews and Meta-Analyses PDF

Evaluating Systematic Reviews and Meta-Analyses Abbey Krysiak, PharmD, BCPP  Review systematic reviews and meta-analysis Differentiate between three types of reviews: narrative (nonsystematic review), qualitative systematic review and quan...

Evaluating Systematic Reviews and Meta-Analyses Abbey Krysiak, PharmD, BCPP  Review systematic reviews and meta-analysis Differentiate between three types of reviews: narrative (nonsystematic review), qualitative systematic review and quantitative systematic review (meta-analysis) Objectives Determine if the results of a systematic review are valid based on a critical evaluation of its methods Interpret results of a meta-analysis Describe heterogeneity and bias in meta-analysis Describe the role of systematic review in research and practice Outline for Today Brief review of systematic reviews and meta-analysis Review interpretation of results from meta-analysis Discuss a published systematic review and meta-analysis – this is different than RCT and cohort! Article posted in Canvas *make sure you take a look at my additional readings for extra fun…* Introduction Systematic review: Qualitative process for identifying and summarizing existing studies that address a specific question Helps us understand the evidence that’s out there This is helpful given the volume of literature and interpreting inconsistent results Meta-analysis: Quantitative synthesis of data derived from individual studies typically identified from a systematic review The aim with a meta-analysis is to produce a single estimate of treatment effect across included studies Systematic Reviews + Meta-Analysis Systematic Reviews Narrative Review: Summary of previously conducted research that lacks systematic methods May address a broad questions rather than focused - do not apply formal criteria for selection of studies included Provide qualitative rather than quantitative information Helpful for obtaining baseline knowledge – tertiary type of literature! Qualitative Systematic Review: Summary that uses specific criteria to address a question but does not statistically combine data Focuses only on relevant studies that will answer a specific question by using inclusion and exclusion criteria (for studies, not patients) Also considered tertiary literature (no new data generated) Sometimes chosen over quantitative because results can’t be pooled – typically would report a range Systematic Reviews Quantitative Systematic Review (Meta-Analysis): Summary of previously conducted studies that statistically combines (i.e. pools) data This is considered primary literature – we are generating new data Typically calculates a single estimate of effect for each endpoint VERY useful when previous studies have been inconclusive or contradictory, or sample size has been too small (power not met) Meta-analyses help with: Supporting or refuting lesser quality evidence Overcoming reduced statistical power of smaller studies Assessing occurrence of rare events Providing guidance with conflicting evidence Displaying sample sizes and treatment effects visually (forest plots) Assessing heterogeneity and publication bias Answering new questions  Traditional Meta-Analysis Most common type – also referred to as pairwise analysis Aggregate data (study level data, summary data) from studies are used to created pooled Meta- estimates to compare two groups Analysis Readily available, but have limitations Each study may handle their data differently – inconsistencies can be introduced For example, if each study defined primary endpoint slightly different, primary endpoint of meta-analysis will include these inconsistencies, which may contribute to heterogeneity Evaluation of Systematic Reviews (Qualitative and Quantitative) Evaluation of Systematic Reviews Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement Guidance document that describes what information should be reported in systematic review Released in 2009 – 27-item checklist (now has other versions for different types of meta-analyses) Evaluation of Systematic Reviews Title, Abstract and Introduction Title should address type of review – not provide results! Abstract should review objective, data sources, eligibility criteria (for studies), synthesis method, main results and conclusion Introduction should summarize previous data on the topic and rationale for the study – objective should be clearly stated here Evaluation of Systematic Reviews Methods and Results Should begin with description of how studies were searched and included Should use all databases: MEDLINE, Embase, others may be used Search terms described and should represent key elements of research question Published and unpublished data may be identified “Grey literature” – data not formally published in standard sources like journals Can be found through clinical trial registries (clinicaltrials.gov), contacting manufacturers and experts in field Important to include to avoid publication bias Evaluation of Systematic Reviews Methods and Results Eligibility criteria: similar to inclusion exclusion criteria → determines which studies will be included in the review Should address PICOTS → population, interventions, comparators, outcomes and timing/setting Overly narrow criteria may lead to omission of relevant studies, whereas overly broad may lead to inclusion of studies that aren’t similar Results of eligibility presented in PRISMA diagram – in addition usually in results, table will summarize characteristics of studies  Methods and Results Assessing quality (i.e. risk of bias) of individual studies is very important Evaluation of Best tool is the Cochrane risk-of-bias – specifically for systematic reviews of RCTs Systematic Involves six domains to assess five types of bias The amount of bias that is acceptable in each Reviews systematic review will vary depending on topic In general, the lower risk of bias → more confidence in results  Methods and Results Nominal data Categories not ranked – yes or no answers Evaluation of Primary measure will be risk ratio (RR), odds ratio (OR) or hazard ratio (HR) Systematic Continuous data Reviews Data are ranked with equal distance between values Measure often reported as difference in means Sometimes, this can be difficult to combine and it will be converted to dichotomous Results typically displayed with a forest plot FOREST PLOT Heterogeneity Assessment of heterogeneity means we are looking at how similar our studies are (eligibility criteria, measurement of outcomes, bias) If they are different – they may not be adequately similar to statistically combine Two ways to assess: Assess the statistical significance: Cochran’s Q test Follows chi-squared and has a p-value P-value >0.10 not statistically significant (desirable) P-value 0.05 would be expected for the plot at the top, indicating that asymmetry was not statistically significant (desirable) and thus publication bias was unlikely pvalue of ≤ 0.05 would be expected for the plot at the bottom, indicating that asymmetry was statistically significant (undesirable) and thus publication bias was likely  Reviews are an essential part of literature Narrative are helpful for general overview of topics Systematic reviews are a critical first step in Summary development of guidelines Meta-analyses can help strengthen the statistical analysis of a systematic review by pooling results These can be considered the highest level of evidence Let’s go through an example! Questions?