Research Methods PDF

Research Methods MODULE 13 Quiz 2 Review Which hypothesis (research vs. null) is involved in Type 1 and Type 2 Errors? What is probability? What is a p value? Systematic Reviews & Meta-analyses Systematic review o Review and summary of studies based on a focused clinical question o If statis...

Research Methods MODULE 13 Quiz 2 Review Which hypothesis (research vs. null) is involved in Type 1 and Type 2 Errors? What is probability? What is a p value? Systematic Reviews & Meta-analyses Systematic review o Review and summary of studies based on a focused clinical question o If statistical methods are used to synthesize the studies, the review is known as a meta-analysis Meta-analysis o Combines the statistical data from all included studies to arrive at a more precise estimate of the effect or outcome Systematic Review: Process A systematic review starts with a clearly focused question called a PICO(T) question o Population (P)  What is the population of interest o Intervention (I)  What is the intervention of interest o Comparison (C)  What will the intervention be compared to  Depending on the study design this step may not apply o Outcome (O)  How will you know if the intervention worked o Time (T), optional Systematic Review: Process Once a PICO(T) question is identified, the researcher needs to: oLocate published (and unpublished, where applicable) consistent with the PICO(T) question oCritically appraise the strength and quality of the evidence in each individual study oSynthesize the overall strengths and weaknesses of the studies in aggregate Systematic Review: Process Once you have identified your studies, the researcher needs to: o Identify the studies to be included based on a rigorous inclusion and exclusion criteria  e.g. including or excluding studies based on disease process, setting, population, geographic location, research method, statistical significance, sample size, etc. o Develop an explicit reproducible methodology to search for, identify, and collect the studies that meet the eligibility criteria  We are trying to be systematic in our process – remember, a research study is only as good as its reproducibility! Systematic Review: Process Once you have identified your inclusion and exclusion criteria and reproducible search methodology: o Critically appraise and synthesize the strength and quality of the evidence in each individual study as well as all the studies as a whole  Some tools available to perform critical appraisal can be reviewed from Module 4  The general critical appraisal questions from Module 10 may also be used quantitative; Module 12 if qualitative Systematic Review: Process Determine the strength and quality of the evidence for practice provided by the chosen study in relation to: o Internal validity o External validity o Sampling styles/concerns o Variable measurements o Data analysis o Applicability of findings to practice (and PICOT question) Evaluation of Research Results: Terminology Used in Meta-Analysis Confidence Interval Likelihood ratio Predictive Value Number needed to treat (NNT) Odds Ratio Relative Risk Sensitivity Specificity  Relative Risk Relative Risk: Risk ratio (RR), or the probability of the outcome for an exposed group versus an unexposed group - Example: Tobacco o What is the risk of developing lung cancer between smokers (exposed) and nonsmokers (unexposed)? - Example: Thyroid cancer o What is the risk of developing thyroid cancer for those in a nuclear radiation zone (exposed) versus 1000 miles away (unexposed) - Presented as a probability, with RR=1 being no difference between groups (a baseline) o 1 is an increased probability Relative Risk Relative Risk Relative Risk Reduction: an estimate of the percentage of baseline risk that is removed as a result of the therapy (intervention) Calculated as the absolute risk reduction (ARR) o ARR = (EER – CER) / CER  EER: experimental group event rate  CER: control group event rate Relative Risk & Relative Risk Reduction Odds Ratio Odds Ratio (OR) o Estimates the odds (probability) of an event occurring relative to the probability of the event NOT occurring o Typically the measurement of choice for nonexperimental studies o OR = 1 means no difference in the groups o OR 1 increased odds in treatment group Odds Ratio Odds Ratio vs. Relative Risk Practice Out of 1000 patients: Lung Cancer No Lung Cancer 250 are smokers, 750 are nonsmokers 81 cases of lung cancer, 57 in smokers and 24 in Smoker A B nonsmokers Calculate the Relative Risk of lung cancer for smokers Nonsmoker C D versus nonsmokers. Calculate the Odds Ratio of lung cancer in smokers versus nonsmokers Sensitivity & Specificity SENSITIVITY o The ability of a test to detect true positives o Few false negatives SPECIFICITY o The ability of a test to detect true negatives o Few false positives Sensitivity and specificity have an inverse relationship: o As sensitivity increases, specificity usually decreases o As specificity increases, sensitivity usually decreases Sensitivity & Specificity Example: If we have 1000 patients and 50% have a disease, we would expect: o A 99% sensitive test to correctly detect 495/500 that have a disease, but we would miss 5 true positives o A 99% specific test to correctly detect 495/500 that do NOT have the disease, but we would miss 5 true negative To calculate sensitivity: True positives / (true positives + false negatives) o 500/(500 + 5) = 99% To calculate specificity: True negatives / (true negatives + false positives) o 500 / (500 + 5) = 99% Sensitivity & Specificity Example: If we have 1000 patients and 50% have a disease, we would expect: o An 80% sensitive test to correctly detect 400/500 that have a disease, but we would miss 100 true positives o A 75% specific test to correctly detect 375/500 that do NOT have the disease, but we would miss 125 true negative To calculate sensitivity: True positives / (true positives + false negatives) o 400 / (400 + 100) = 80% To calculate specificity: True negatives / (true negatives + false positives) o 375/ (375 + 125) = 75% Sensitivity & Specificity: Practice To calculate sensitivity: True positives / (true positives + false negatives) To calculate specificity: True negatives / (true negatives + false positives) We have 100 patients waiting in the ED with complaints of chest pain. We draw a high-sensitivity troponin level on all 100 patients. 25 have a positive troponin level, but determine that only 12 individuals meet criteria for an NSTEMI. 1 individual did have an NSTEMI, but the troponin was not elevated and we missed it. What is the sensitivity and specificity of this test? Sensitivity: Specificity: Predictive Value Predictive value is how clinicians USE sensitivity and specificity Positive predictive value: o How likely is it that this patient has the disease given that the test result is positive? o True positives / (true positives + false negatives) Negative predictive value: o How likely is it that this patient does not have the disease given that the test result is negative? o True negatives/ (true negatives + false positives) In the clinical setting, these concepts are dependent on o The population being tested AND o The prevalence of a disease process Predictive Value Likelihood Ratio Likelihood ratio is a measure that a given test result would be expected in a patient with the target disorder compared with the likelihood that that same result would be expected in a patient without the target disorder o Measures the power of a test to change the pre-test into the post-test probability of a disease being present o Likelihood ratios can be positive or negative o Larger likelihood ratios provide higher degrees of certainty A large positive likelihood ratio provides APRNS with a high degree of certainty that the patient has the disease A large negative likelihood ratio provides APRNs with a high degree of certainty that the patient does NOT have the disease A small likelihood ratio in either direction provides little to no change in the degree of certainty Predictive Value vs. Likelihood Ratio Predictive Value changes based on your population o Probability that a positive test indicates a positive disease state  A lung nodule on a CT scan in a nonsmoker is less likely to represent lung cancer than it is in a smoker Likelihood ratio does NOT change based on the population o Probability of a positive test result in someone with the disease by the probability of a positive test result in someone without the disease.  A flu test is very likely to indicate active influenza infection regardless of population Number Needed to Treat (NNT) Measure for determining intervention effectiveness and its application to individual patients The number of people who need to receive a treatment/intervention for one patient to receive a benefit o Lower NNTs are associated with cost-effective interventions o High NNTs are associated with increased costs and relatively lower efficacy NNT places an emphasis on: o Utilitarianism o Cost effectiveness https://clincalc.com/stats/nnt.aspx What Healthy People 2030 guidelines can you think of where the principles of NNT might apply?  Confidence Intervals (CI) Range of values based on any given sample of the population The likelihood that a population parameter will fall within a certain range on repeat testing 95% is the most commonly accepted CI because (in a normal distribution of values) it contains all values within two standard deviations of the population mean Systematic Review: Process A systematic review will typically include a PRISMA diagram o Illustrates flow of information through the phases of the systematic review o Maps out the number of studies included and excluded and the inclusion/exclusion criteria Systematic Review: Process After studies in a systematic review are retrieved, assessed, and synthesized, practice recommendations are made and presented The components of a systematic review are the same for a meta-analysis except a meta-analysis quantitatively evaluates the impact of the studies as a whole Types of Systematic Reviews o Meta Analysis o Integrative o Rapid o Scoping o Narrative o Scoping o Realist Meta-Analysis Uses statistical techniques to assess and combine studies of the same design to obtain a precise estimate of effect Provides Level I evidence Determines the impact of a number of studies Two phases o Phase I: Data extraction o Phase II: The decision is made as to whether it is appropriate to calculate a pooled average result (effect) of the studies reviewed Meta-Analysis Effect size o Calculated using the difference in the average scores between the intervention and control groups from each study and from each of the studies reviewed to obtain an estimate of the population (or the whole) to create a single effect size of all the studies Forest plot (blobbogram) o Graphically depicts the results of analyzing a number of studies Integrative Review Critical review of an area of research without a statistical analysis or a theory synthesis Broadest category of review May include: o Theoretical literature and/or research literature o Methodology studies, theory review, or the results of differing research studies o Quantitative or qualitative research Statistics are not used to summarize and generate conclusions about the studies. Integrative Review Although systematic reviews are highly useful, they also must be reviewed for potential bias and carefully critiqued for scientific rigor. Other review types not based on a statistical analysis basically follow the same systematic process but have slightly different foci and can be titled: o Narrative o Scoping o Rapid o Realist Reviews Cochrane Collaboration Cochrane collaboration/review o Largest repository of meta-analyses o Prepares and maintains a body of systematic reviews that focus on health care interventions o Collaborates with a wide range of health care professionals for developing reviews Systematic Review Tools PRISMA MOOSE: Meta-analysis of Observational Studies in Epidemiology Clinical Practice Guidelines Clinical practice guidelines o Systematically developed statements or recommendations that link research and practice and serve as a guide for practitioners Evidence-based practice guidelines o Developed using a scientific process Expert-based practice guidelines o Depend on a group of nationally known experts who meet and use opinions of experts along with whatever research evidence is developed to date  Critical Appraisal: EBP & Clinical Guidelines Does the PICO question match the review studies? Is the date of publication or release current? Are the review methods clearly stated and Are the authors of the guideline clear and appropriate? comprehensive? Are the dates of the review’s inclusion clear and Is the clinical problem and purpose clear? relevant? What types of evidence were used? Are the inclusion and exclusion criteria clear? Is there a description of the evidence grading methods? What criteria were used to assess each of the studies? Were the search terms and retrieval methods clear? If studies were analyzed individually, were the data clear? Is the guideline well-referenced and comprehensive? Were the methods of study combination clear? Are the recommendations sourced to the level of If reviewed collectively, how large was the effect? evidence? Are the clinical conclusions drawn from the studies Has the guideline been reviewed by experts? relevant and supported by the review? Who funded the guideline development? Developing EBP: Clinical Inquiry Strategy 1: Asking a focused clinical question o Develop the question by addressing these four issues:  1.Population  2.Intervention  3.Comparison  4.Outcome Assign the question to a clinical category Developing EBP: Clinical Inquiry ASSIGNING A CLINICAL CATEGORY Prognosis category Therapy category o Nonexperimental o Experimental or quasi-experimental o Follow-up o Outcome known o Determination of factors o Therapy appraisal tool o Prognosis tool Diagnosis category Harm category o Cross-sectional o Nonexperimental o Comparison of the new and the “gold standard” o Exposure o Diagnostic tool o Harm appraisal tool Developing EBP: Literature Search Factors affecting a literature search˜ oInformation literacy oConsult librarian oElectronic indexes/databases oWeb-based tutorials to search databases Screening your results oPeer-reviewed journal? oSimilar setting and sample? oStudy sponsorship? Developing EBP: Critical Appraisal Therapy Category (experimental, RCTs, intervention studies) o Is there a difference between two or more interventions?  Numerical values are either continuous or discrete o Measure outcomes using discrete variables  Present results as measures of association  Relative risk (RR) Relative risk reduction  Odds ratio (OR)  Number needed to treat (NNT)  Confidence interval (CI) Null value Developing EBP: Critical Appraisal Diagnosis articles oSensitivity (True positive) oSpecificity (True negative) oPositive predictive value (PPV) oNegative predictive value (NPV) oLikelihood ratio (LR) Positive or Negative Developing EBP: Critical Appraisal Prognosis articles o Odds ratio (OR)  Probability of an event  Calculated by dividing the odds in the treated or exposed group by the odds in the control group Developing EBP: Critical Appraisal Harm articles o Used to determine if an individual has been harmed by being exposed to a particular event. o Case-control design: investigators select the outcome they are interested in and examine if any one factor explains those who have and do not have the outcome of interest. o The measure of association that best describes the analyzed data in case- control studies is the odds ratio. Application: Meta-analysis to EBP Meta-analysis o A research method that statistically combines the results of multiple studies (usually randomized clinical trials) to answer a focused clinical question through an objective appraisal of carefully synthesized research evidence o Uses a clinical question to guide the process. o All relevant studies are gathered o At least two people assess the quality of each study o Results of individual studies are statistically combined Evidence-based practice o Integrating individual clinical expertise and patient preferences with the best external evidence to guide clinical decision making o Day-to-day practice can be more evidence based with practice and a few simple tools Developing EBP Use of evidence-based practices is an expected standard Implementation requires the use of strategies that address systems of care, individual practitioners, leadership, and health care cultures Translation of research into practice (TRIP) o Multifaceted, systemic process of promoting adoption of EBPs in delivery of health care services that goes beyond dissemination of evidence-based guidelines Which of these are AGCNS competencies? EBP and the AGCNS: Core Competencies Conduct of research o Systematic investigation of a phenomenon to address research questions and test hypotheses that advance the state of the science Evidence-based practice o Conscientious and judicious use of current best evidence in conjunction with clinical expertise and patient values and circumstances to guide health care decisions Translation science (implementation science) o Focuses on testing implementation interventions to  Improve uptake and use of evidence to improve patient outcomes and population health  Identify what implementation strategies work for whom, in what settings, and why EBP: The Iowa Model The Iowa Model o Clinical question o Literature review/study conducted o Recommendations are based on evidence synthesis o Findings are compared with current practice, and a decision is made about the necessity for a practice change o A pilot evaluation is conducted and EBP is refined o The change is implemented, and outcomes are monitored The Iowa Model Using the Iowa Model  Selection of a topic  Problem-focused triggers  Knowledge-focused triggers  Forming a team  Interdisciplinary  Include stakeholders  Evidence retrieval  Relevant research  Related literature  Classification of articles Using the Iowa Model  Read and critique articles in order  Clinical articles to understand the state of the practice  Theory articles to understand the theoretical perspectives and concepts that may be encoun-tered in critiquing studies  EBP guidelines and evidence reports  Systematic reviews, meta-analyses, and synthesis reports to understand the state of the sci-ence  Primary research articles Using the Iowa Model  Schemas for grading the evidence  Quality & Strength  Critique and synthesis of research  Group approach recommended Distributes the workload Helps in understanding the scientific base for the practice change Arms nurses with citations and research-based language Provides novices an environment to learn critique Using the Iowa Model  Decide which studies to use based on overall scientific merit, type of subjects enrolled, and relevance to the topic  Summary table essential information  Research questions/hypotheses  Independent and dependent variables studied  Description of the study sample and setting  Type of research design  Methods used to measure variables and outcomes  Study findings Using the Iowa Model  Setting forth evidence-based practice recommendations  Based on the critique of practice guidelines and synthesis of research  Decision to change practice  Relevance of evidence for practice  Consistency in findings  Similar sample characteristics  Consistency among evidence  Feasibility for use in practice  Risk/benefit ratio and cost Using the Iowa Model  Development of evidence-based practice  Document the evidence base of the practice using the agreed-upon grading schema  Provide key stakeholders with an opportunity to review and provide feedback  Implementing the practice change  Trip Model  Nature of the innovation  Communication to members of a social system Using the Iowa Model  Implementation strategies to address the characteristics of the EB topic  Relative advantage of the EBP  Compatibility with values, norms, work, and perceived needs of users  Complexity of the EBP topic Using the Iowa Model  Strategies to promote adoption  Quick reference guides  Computerized clinical decision support (CDS) tools  Key messaging at the point of care delivery  Strategies to address communication  Social and mass media  Education  Opinion leadership  Change champions  Educational outreach/academic detailing Using the Iowa Model  Strategies to address users  Audit and feedback  Performance gap assessment (PGA)  Trying the EBP  Engaging with recipients of the EBPs  Implementation team meetings to address barriers and acknowledge success Using the Iowa Model  Strategies to address the social system  Conducting an environmental scan  Meeting with key leadership stakeholders  Revising practice standards and documentation systems  Providing recognition and rewards Using the Iowa Model  Evaluation  Structured approach to evaluate the impact of the EBPs that are implemented  Key component of piloting the change in practice  Includes both process and outcome measures Evidence-Based Practice & the AGCNS Challenges to meet EBP expectations o Building organizational capacity o Education of clinicians and leaders to foster use of EBPs o Engaging patients and families as part of the implementation team An emerging body of knowledge in translation science provides an empirical foundation for implementation strategies to assure delivery of high quality, evidence-based care