Chapter17.pdf

Chapter 17 Learning From Systematic Reviews Research Integration and Synthesis #1 The systematic and rigorous integration and synthesis of evidence is a cornerstone of evidence- based practice (EBP). Impossible to develop “best practice” guidelines, protocols, and procedures wi...

Chapter 17 Learning From Systematic Reviews Research Integration and Synthesis #1 The systematic and rigorous integration and synthesis of evidence is a cornerstone of evidence- based practice (EBP). Impossible to develop “best practice” guidelines, protocols, and procedures without organizing and evaluating research evidence through a systematic review Copyright © 2022 Wolters Kluwer. All Rights Reserved. Research Integration and Synthesis #2  Forms of systematic reviews o Narrative, qualitative integration (traditional review of quantitative or qualitative results) o Meta-analysis (statistical integration of results used to compute common effect size) o Metasynthesis (theoretical integration and interpretation of qualitative findings) Copyright © 2022 Wolters Kluwer. All Rights Reserved. Meta-Analysis: Advantages Objectivity—statistical integration eliminates bias in drawing conclusions when results in different studies are at odds. Increased power—reduces the risk of a Type II error compared to a single study Despite these advantages, meta-analysis is not always appropriate. Indiscriminate use has led critics to warn against potential abuses. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Criteria for Using Meta-Analysis in Systematic Review  Must decide whether statistical integration is suitable  Research question or hypothesis should be essentially identical across studies. o Avoid the “fruit” problem—don’t combine apples and oranges!  Must have a sufficient knowledge base—must be enough studies of acceptable quality  Results can be varied but not totally at odds. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Question #1 Tell whether the following statement is True or False. An advantage of meta-analysis is the conclusions arrived at. a. True b. False Copyright © 2022 Wolters Kluwer. All Rights Reserved. Answer to Question #1 a. True Rationale: Meta-analysis offers a simple advantage as an integration method: objectivity. Readers of a meta- analysis can be confident that another analyst using the same data set and analytic decisions would reach the same conclusions. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Steps in a Meta-Analysis #1  Problem formulation: Delineate research question or hypothesis to be tested.  Design of meta-analysis: Identify sampling criteria for studies to be included.  Search for evidence in literature: Develop and implement a search strategy.  Evaluation of study quality: Locate and screen sample of studies meeting the criteria.  Extraction and encoding data for analysis: Create a dataset. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Steps in a Meta-Analysis #2  Calculation of effects: Calculate an effect size (ES) index.  Data analysis: Determine the weighted average.  Assessment of degree of confidence: GRADE Copyright © 2022 Wolters Kluwer. All Rights Reserved. Search Strategy  Identify electronic databases to use.  Identify additional search strategies (e.g., ancestry approach).  Decide whether or not to pursue the grey literature (unpublished reports).  Avoid publication bias.  Identify keywords for the search. Think creatively and broadly. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Approaches to Evaluating Study Quality #1 Meta-analysts make decisions on handling study quality. Approaches o Omit low-quality studies (e.g., in intervention studies, non-RCTs) o Give more weight to high-quality studies. o Analyze low- and high-quality studies to see if effects differ (sensitivity analyses). Copyright © 2022 Wolters Kluwer. All Rights Reserved. Approaches to Evaluating Study Quality #2  Evaluations of study quality can use: o A scale approach (e.g., use a formal instrument to “score” overall quality) o A component approach (code whether certain methodological features were present or not, e.g., randomization, blinding, low attrition) Copyright © 2022 Wolters Kluwer. All Rights Reserved. Question #2 After identifying the research question to be tested for a meta-analysis, what task would the researcher complete next? a. Develop a search strategy. b. Locate sample of studies. c. Identify sampling criteria. d. Extract data from reports. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Answer to Question #2 c. Identify sampling criteria. Rationale: Once the research question has been delineated, the next step is to identify sampling criteria for studies to be included. Then, the researcher develops and implements a search strategy, locating and screening the sample of studies that meet the criteria. Next, the researcher appraises the quality of the study evidence and extracts and records data from the reports. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Analytic Decisions in Meta-Analysis  What effect size index will be used?  How will heterogeneity be assessed?  Which analytic model will be used?  Will there be subgroup (moderator) analyses?  How will quality be addressed?  Will publication bias be assessed? Copyright © 2022 Wolters Kluwer. All Rights Reserved. Heterogeneity #1 Results (effects) inevitably vary from one study to the next. Major question: Is heterogeneity just random fluctuations? o If “yes,” then a fixed effects model of analysis can be used. o If “no,” then a random effects model should be used. Heterogeneity can be formally tested but also can be assessed visually via a forest plot. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Heterogeneity #2 Factors influencing variation in effects are usually explored via subgroup analysis (moderator analysis). Do variations relate to: o Participant characteristics (e.g., men vs. women)? o Methods (e.g., RCTs vs. quasi-experiments)? o Intervention characteristics (e.g., 3-week vs. 6- week intervention)? Copyright © 2022 Wolters Kluwer. All Rights Reserved. Question #3 Tell whether the following statement is True or False. A key component of meta-analysis is the calculation of an effect size index. a. True b. False Copyright © 2022 Wolters Kluwer. All Rights Reserved. Answer to Question #3 a. True Rationale: An effect size index is a central feature of meta-analysis. It is computed for each study and then combined and averaged. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Metasynthesis  Terminology relating to qualitative integration is diverse and complex.  Metasynthesis is an umbrella term, broadly representing “a family of methodological approaches to developing new knowledge based on rigorous analysis of existing qualitative research findings.”  Metasynthesis is not a literature review.  Integrations that are more than the sum of the parts —novel interpretations of integrated findings Copyright © 2022 Wolters Kluwer. All Rights Reserved. Metasynthesis: Steps  Similar to meta-analysis in many ways o Formulate problem. o Decide on design: selection criteria, search strategy. o Search for data in the literature. o Evaluate study quality. o Extract data for analysis. o Perform data analysis and interpretation. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Metasynthesis Approaches #1 Noblit and Hare (developed an approach for a meta-ethnography) o Suggests a seven-phase approach o Involves “translating” findings from qualitative studies into one another o “An adequate translation maintains the central metaphors and/or concepts of each account.” o Final step is synthesizing the translations. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Metasynthesis Approaches #2 Sandelowski and Barroso’s approach distinguishes studies that are summaries (no conceptual reframing) and syntheses (studies involving interpretation and metaphorical reframing). Both summaries and syntheses can be used in a meta-summary, which can lay a foundation for a metasynthesis. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Question #4 Tell whether the following statement is True or False. According to Sandelowski and Barroso, a meta- summary lays the foundation for meta-analysis. a. True b. False Copyright © 2022 Wolters Kluwer. All Rights Reserved. Answer to Question #4 b. False Rationale: A meta-summary, as described by Sandelowski and Barroso, lays the foundation for a metasynthesis. Copyright © 2022 Wolters Kluwer. All Rights Reserved. Meta-Summaries Involve making an inventory of findings and can be aided by computing manifest effect sizes (effect sizes calculated from the manifest content in the studies in the review) Two types o Frequency effect size o Intensity effect size Copyright © 2022 Wolters Kluwer. All Rights Reserved. Effect Sizes in Meta-Summaries Frequency effect size o Count the total number of findings across all studies in the review (specific themes or categories). o Compute prevalence of each theme across all reports (e.g., the #1 theme was present in 75% of reports). Intensity effect size o For each report, compute how many of the total themes are included (e.g., report 1 had 60% of all themes identified). Copyright © 2022 Wolters Kluwer. All Rights Reserved. Sandelowski and Barroso’s Metasynthesis  Can build on a meta-summary  But can only be done with studies that are syntheses (not summaries) because the purpose is to offer novel interpretations of interpretive findings —not just summaries of findings Copyright © 2022 Wolters Kluwer. All Rights Reserved.