Calibrate Dx: A Guide to Improve Diagnostic Decisions PDF

Calibrate Dx: A Resource To Improve Diagnostic Decisions e PATIENT SAFETY  1 Calibrate Dx: A Resource To Improve Diagnostic Decisions...

Calibrate Dx: A Resource To Improve Diagnostic Decisions e PATIENT SAFETY  1 Calibrate Dx: A Resource To Improve Diagnostic Decisions Prepared for: Prepared by: Agency for Healthcare Research and Quality Center for Innovations in Quality, Effectiveness, and Safety U.S. Department of Health and Human Services (IQuESt), Michael E. DeBakey Veterans Affairs Medical 5600 Fishers Lane Center and Baylor College of Medicine, Houston, TX Rockville, MD 20857 Co-Leads: Andrea Bradford, Ph.D., and Ashley N.D. www.ahrq.gov Meyer, Ph.D. Ashish Gupta, M.D., M.B.A. Contract No. HHSP2332015000221/75P0011 Hardeep Singh, M.D., M.P.H. 9F37006 Task Order 5a Supported by: This project was funded under contract HHSP23320150 MedStar Institute for Quality and Safety, MedStar Health, 0022I/75P00119F37006 to MedStar Health Institute for Columbia, MD Quality and Safety (MIQS) from the Agency for Healthcare Kisha Ali, M.S., Ph.D. Research and Quality (AHRQ), U.S. Department of Health Christine Goeschel, Sc.D., R.N. and Human Services. The authors are consultants to MIQS and are solely responsible for this document’s contents, Logistical Support: findings, and conclusions, which do not necessarily Alexis Offner, M.P.H., Michael E. DeBakey VA Medical represent the views of AHRQ. Readers should not interpret Center and Baylor College of Medicine any statement in this product as an official position of AHRQ or of the U.S. Department of Health and Human Services. Acknowledgments This resource was developed collaboratively with a team Consultants of subject matter experts who contributed content, case Gopi J. Astik, M.D., Northwestern University examples, and recommendations. We thank the following Elisabeth Belmont, J.D., MaineHealth individuals for their substantive and valuable contributions Melanie A. Grady, M.D., MedStar Health to the resource: Joseph A. Grubenhoff, M.D., Children’s Hospital Colorado Jonathan S. Ilgen, M.D., Ph.D., M.C.R., Subject Matter Experts University of Washington Medical Center Susrutha Kotwal, M.B.B.S., M.D., Johns Hopkins University and Harborview Medical Center Kathleen Lane, M.D., University of Minnesota Zaven Sargsyan, M.D., Baylor College of Medicine and Geeta Singhal, M.D., M.Ed., FAAP, Baylor College of Michael E. DeBakey VA Medical Center Medicine and Texas Children’s Hospital Robert L. Trowbridge, M.D., Tufts University and Maine Medical Center AHRQ Publication No. 22-0047-2-EF September 2022  2 Introduction “Better is possible. It does not take genius. It takes diligence. It takes moral clarity. It takes ingenuity. And above all, it takes a willingness to try.”1 - Atul Gawande Lifelong learning is essential for achieving and maintaining diagnostic excellence. Diagnostic excellence involves not just making a correct and timely diagnosis but also doing so while using the fewest resources, optimizing patient experiences, and managing uncertainty.2 Errors in diagnosis likely affect more than 12 million U.S. adults per year (or 1 in 20), and about half of these errors can lead to severe or permanent harm.3 Reducing these errors requires both individual clinicians and healthcare systems to work toward diagnostic excellence.4 Calibration, defined as alignment between a person’s diagnostic accuracy and their confidence in that accuracy, is an essential component of diagnostic excellence.2 Miscalibration, i.e., misalignment between a person’s diagnostic accuracy and their confidence in that accuracy, can manifest as either overconfidence or underconfidence (Figure 1). Miscalibration can lead to misdiagnoses, delayed diagnoses, under- or overtesting, and inefficient diagnostic processes.5 Figure 1. Calibration Alignment and Effects on Diagnostic Decision Making WELL-ALIGNED UNDERCONFIDENCE CALIBRATION OVERCONFIDENCE Inefficient use of Achieves diagnostic excellence Increased likelihood resources (testing, of missed or wrong consultations) Judiciously uses testing and diagnosis referrals Potential for unnecessary Missed opportunity to delays in diagnosis Knows where to focus enhance clinical expertise and treatment continuing education and improvement efforts  3 Once clinicians complete their professional training and begin to practice independently, they seldom receive formal feedback on their diagnostic processes and outcomes. This situation creates a “calibration gap”6 or a gap between how they think they are doing and how they are actually doing. As such, clinicians may underestimate the number of their patients who experience a missed, delayed, or wrong diagnosis.7,8 Timely and effective feedback on clinicians’ diagnostic decisions can fill this gap.9,10 This resource aims to support clinicians like you in your quest for diagnostic excellence by providing tools to address the “calibration gap.” This resource will guide you through a series of steps to self-assess and generate feedback about your diagnostic decision making and use that information to help you become better calibrated. When you compare your perceived performance to your actual performance, you will make better decisions about what is going well and where you should focus your learning and improvement efforts. Getting Started WHO SHOULD USE THIS RESOURCE? This resource is primarily aimed at individual clinicians whose scope of practice includes diagnosis. However, secondary potential audiences can include any learners, educators, or health professionals in medicine. SCOPE OF THIS RESOURCE This resource focuses on diagnosis, including processes involved in making a diagnosis and the outcome of giving an explanatory label to patients after these processes unfold. Although processes related to management and treatment of patients sometimes overlap, the context for this resource is exclusively diagnosis. WHAT YOU WILL NEED TO USE THIS RESOURCE Dedicated time. We estimate this self-evaluation exercise will require a few hours of your time; we recommend repeating the exercise every few months to maximize your learning. Your own cases to review. One or more partners with whom to discuss your learning. PRIVACY, SECURITY, CONFIDENTIALITY, AND PRIVILEGE CONSIDERATIONS Check with the appropriate point of contact in your organization before creating a list of cases, accessing your patients’ records, or beginning to work with the tools provided for this activity. Having access to medical records for patient care purposes does not necessarily mean that you or your colleagues are authorized to access your patients’ information for other purposes. Also, be aware that if confidentiality and privilege protections for this kind of activity are available and desired, it is likely that they will only apply if specific requirements are followed. Certain steps may need to be taken in advance and the activity may need to be conducted in a certain way. In addition, information related to this activity may need to be stored in a certain location to ensure compliance with the Health Insurance Portability and Accountability Act Privacy and Security Rules and any requirements related to confidentiality and privilege protections. You should also be clear about how information related to the activity might be shared and used within your organization and for what purposes.  4 Overview of Calibration Exercise This resource will guide you in implementing four steps of the diagnostic calibration learning and improvement cycle shown in Table 1. The sections that follow further describe each step in the cycle. TABLE 1. Diagnostic Calibration Learning and Improvement Cycle GUIDE STEP ACTIVITY REFERENCE Choose an area of practice for which you would like to be better calibrated. You will likely learn SPECIFY the calibration task Page 5 more by focusing on a specific area of practice than reviewing your cases at random. EVALUATE diagnostic performance using Select a small sample of your cases, review self-assessment and peer feedback tools in them for learning opportunities, and seek further Page 7 this resource feedback from a colleague. Identify improvement strategies for yourself (and, PLAN AND APPLY improvement when appropriate, your team and your system), strategies and continuously monitor Page 10 and begin to take appropriate action. Repeat the performance previous steps at regular intervals. Reflect on this calibration exercise over time, REFLECT on this exercise and adjust if evaluate additional areas of interest, and make Page 13 needed adjustments as needed. *Ambrose SA, et al. How Learning Works. San Francisco: Jossey-Bass; 2010.11 If you are using this resource for the first time, review each of the following sections to understand these steps in the calibration learning and improvement cycle. You can navigate back to these sections as needed using the table of contents. To see an example of how these steps might look in practice, see “Putting It All Together” (page 14).  5 Specify The Calibration Task MAIN IDEA Decide what areas are important to your clinical practice and focus on case scenarios that you encounter fairly often. Table 2 lists certain types of scenarios that you might consider Tip: As a general rule, focus on for calibration exercises. Calibration exercises should occur at diagnoses and situations that are regular intervals so that you can monitor your clinical reasoning common in your practice, rather than and diagnostic outcomes over time. rare events. Table 2. Case Scenarios To Consider for Calibration Exercises TYPE OF SCENARIO WHY CONSIDER IT EXAMPLES Diagnosis-specific Certain diagnoses are known to be Sepsis, meningitis, stroke, appendicitis, cancer, situations frequently missed or delayed. pulmonary embolism Undifferentiated Common symptoms with broad differential Abdominal pain, shortness of breath, presentations diagnoses may be susceptible to cognitive abnormal uterine bleeding biases. Unexpected trajectories Specific events suggest presence of an Patients with change in diagnosis or management earlier opportunity to make the correct during emergency room visit or hospitalization, diagnosis. unexpected escalation of care or return visit, repeat visits for the same condition/concern without a definitive diagnosis Diagnostic test Certain lab and imaging studies and other Investigations for anemia, assessment of cognitive interpretation or followup tests are prone to misinterpretation or impairment, pulmonary nodules on chest x ray missed followup. High-risk situations Specific situations, such as care transitions Clinician handoffs, transfers between hospitals, or complex cases involving multiple incidental findings, multimorbidity, disciplines, are known to be at high risk for multiple subspecialists involved in patient’s care error. Patient populations at Patients who face systemic social and People with undocumented status, those who are higher risk health inequities may be at higher risk for uninsured or underinsured, those facing challenges breakdowns in care. related to health literacy  6 Formulating Questions To Guide Calibration Once you have chosen a general area of focus, consider both the diagnostic processes13,14 and diagnosis-related outcomes15 for which you will need more information to evaluate your performance. For instance, you can focus on steps in the diagnostic process (e.g., could I improve certain aspects of the workup I did to make the diagnosis, regardless of its accuracy?). Another option is to compare your perceived outcomes with actual outcomes (e.g., how appropriate was my diagnostic assessment? Was my diagnosis correct?), as shown in Table 3. The standards by which you evaluate your diagnostic reasoning and decision making can depend on several factors, including the availability of a reference standard or clinical guidelines. In many situations, no clear “gold standard” workup or assessment strategy exists. Thus, different clinicians are likely to anticipate different risks, use resources in different ways, ask for help differently, and monitor a situation differently. The goal is not necessarily to identify a single, correct process or diagnosis (there might not be one). It is to understand if the process you used and diagnosis you made were reasonable given the information available to you at the time—that is, would most reasonable clinicians in the same situation make the same diagnostic decisions? Develop one or more calibration questions to guide your review as you examine your cases. Focus on areas of improvement that are most important to you. Examining both processes and outcomes may help you evaluate your performance and become better calibrated. The examples below are to stimulate your thinking and are neither exhaustive, nor prescriptive. Table 3. Sample Calibration Questions for Diagnostic Process and Outcome Domains DIAGNOSTIC PROCESS DIAGNOSTIC OUTCOMES Domain Sample Question Domain Sample Question Patient-provider encounter Was the differential diagnosis Effectiveness What was the patient’s ultimate (e.g., history and physical sufficiently broad? diagnosis and how did I make it? examination) Diagnostic test Were the ordered tests Timeliness Could I have made a correct diagnosis performance and indicated by the clinical sooner? interpretation situation? Followup and tracking of Did I follow up on labs, imaging Efficiency Were the time and resources I used to diagnostic information studies, and consultant arrive at the diagnosis more or less than recommendations in a timely I expected? manner? Subspecialty consultations Was the amount of workup Safety Did a knowledge gap, cognitive bias, or and referrals adequate before consulting or problem with attention or memory lead referring to a specialist? to missing important findings? Patient factors/behaviors Did barriers to communicating Patient Did I communicate the diagnosis to the effectively with the patient centeredness patient effectively and in a timely change my diagnostic process? manner? Equity Were my outcomes consistent across patients of all backgrounds? Note: Diagnostic process questions are organized according to a conceptual model of diagnosis from the National Academies of Sciences, Engineering, and Medicine.14 The outcomes are organized according to the six domains of healthcare quality.15  7 Evaluate Diagnostic Performance MAIN IDEA A systematic approach to reviewing the care of your patients will help you identify the information you need to evaluate your diagnostic decision making and facilitate improved calibration. Including a colleague in this process will add value. SELECT CASES FOR REVIEW AND REFLECTION Choose three to five cases to review to assess your performance. If you need to narrow down a larger list of cases, opt for more recent cases as you may remember more clearly the circumstances and your mindset during those cases. Aim for a variety of cases over time, not limited to those that are unusual or highly memorable. Do not limit the cases you choose to times when things went wrong. If you can select cases independent of the outcome, you are more likely to understand your typical performance. Define objective selection criteria to obtain a more representative sample. Examples of selection criteria include: A random sample of all patients you diagnosed with pneumonia in the past 6 months. The last few patients who presented with abdominal pain. The last few patients who were unexpectedly admitted (or readmitted) to the hospital after an appointment with you. Patients transferred to another clinician’s care whose diagnosis subsequently changed significantly. A random sample of patients with whom you experienced communication barriers. Your primary sources of information will be the medical record and your recollection of the case, stimulated by review of clinical documentation. Secondary sources of information may include followup with the patient (when appropriate and allowable) and other involved clinicians. WHAT DEFINES A “CASE”? To evaluate diagnostic performance, focus on the initial patient presentation and the subsequent trajectory related to the same condition. Depending on the setting, the diagnostic process may unfold over multiple days (or longer) and multiple encounters. Thus, before reviewing a case, you will need to ensure you can access all the relevant information needed to review the evolution of the patient’s diagnosis. HOW TO FIND CASES THAT MEET YOUR CRITERIA Create a prospective followup list. Many electronic health records (EHRs) allow clinicians to create personal reminders or lists. As you encounter patients who would be appropriate to follow up for calibration activities, include them in this list – if you have confirmed it can be used for this purpose. If you have a reminder function, you can use it to specify a time to review the record (e.g., 3 months after initial diagnosis). To avoid hindsight bias (i.e., when knowing an outcome overly influences your evaluation of the processes leading to the outcome16), consider adding cases to your list before outcomes are known, and then follow up at a later date. Work with informatics personnel to identify relevant cases. Some EHR systems include query tools that can generate a list of cases that meet criteria you specify, such as date of service and diagnosis. A health information technology or informatics specialist in your organization may be able to create a report (e.g., patients you diagnosed with a certain condition in the last month) that provides a list of cases relevant to your calibration goals.  8 Learn From Your Own Cases Once you have identified cases, review each case individually and RECOMMENDED TIME then reflect on your reasoning across cases. Use the Diagnostic COMMITMENT Calibration Debrief Tool (Appendix A) as a guide. Follow the steps below. 3-4 hours CONSIDER USING A STANDARDIZED TOOL FOR Once per quarter INDIVIDUAL CASE REVIEWS Your calibration questions may require collecting diagnosis-specific information from medical records. However, the Revised Safer SET ASIDE A DEDICATED Dx Instrument and Fishbone diagram tools may help you work TIME AND SPACE FOR through the assessment of your diagnostic reasoning and identify YOUR REVIEW opportunities for improvement for each case (see Appendixes for Completing the review and examples): reflection process at one time is The Revised Safer Dx Instrument (Appendix B) is a validated tool recommended. Ensure you have consisting of 13 items that prompt review of the diagnostic reliable access to patient records. process.17 This tool can help you determine whether you had any If applicable, consider having missed opportunities in the diagnostic process. Instructions for practice guidelines, evidence- using the instrument are included in the open-access publication based medicine references, and about development of this tool17 and in the appendix. other relevant materials ready to A Fishbone diagram (Appendix C), modified for diagnostic safety access. events, can be used to break down complex events according to different types of contributing factors, including system-related WHY WORK WITH ANOTHER and cognitive factors. Instructions for applying a fishbone diagram CLINICIAN? to diagnostic safety are discussed in an open-access paper by A colleague can be a helpful source Reilly, et al.18,19 of feedback, especially when no clear standard for calibration is SUMMARIZE YOUR ASSESSMENT OF YOUR DIAGNOSTIC PERFORMANCE IN A BRIEF NARRATIVE available. Your colleague might also provide useful feedback that Develop an overall self-assessment of your performance across can help prevent “overcorrecting” cases. You might note specific aspects of the diagnostic process in response to what you that went especially well, those that could be improved, and learn. Reviewing each other’s contextual factors that might have influenced the diagnostic process. performance can be mutually Summarize the “take home” message and consider action steps beneficial and enhance learning for that may enhance your performance. Section 2 of the Diagnostic both. Calibration Debrief Tool can facilitate this brief write up. The section “Putting It All Together” provides an example of how a clinician can approach this exercise. Appendixes D-F provide further examples of completed case review tools in other practice areas.  9 MEET WITH A COLLEAGUE FOR A “DEBRIEF” Share your self-assessment narrative with one or more colleagues to solicit feedback. For example, ask a peer at a similar career stage or skill level or a more experienced clinician to review it. If permissible and feasible, arrange to make your source data (i.e., medical records) accessible to your colleagues. While agreement is neither the goal nor an assurance of accuracy, a difference of opinion could provide a clue that something could have been done differently and should be investigated further. The goal is to discuss the actions and thought processes involved in a particular patient care situation, encourage reflection on those actions and thought processes, and incorporate improvement into future performance.20 Consider working collaboratively so that you and your colleagues can share and learn from one another. It may be helpful to walk through each case by reflecting aloud, starting with “Tell me about the case” and then asking followup questions such as “Take me into the room. What were you worried about?”…“What did you do next?”…“What did you think might happen?”…“When/how did you decide to ask someone for help?”…“How did you monitor the situation? What were you looking for?” Questions about how you managed uncertainty may yield further useful insight into each other’s decision making. NOTE: During record reviews, you may find cases in which a patient experienced or is at risk of harm that was not previously recognized or disclosed to the patient. In these situations, action may be needed to ensure the patient is safe and that the appropriate parties in the organization are aware so they can take any necessary actions. For example, your organization may have a Communication and Resolution Program (CRP) process that should be initiated. Follow your organization’s policies to notify the appropriate individual/office (e.g., quality or risk management program) of any newly identified harm or risk of harm to a patient.  10 Plan and Apply Improvement Strategies MAIN IDEA Act on what you found by identifying improvement opportunities. These may include ways of building your own knowledge and skills, but also consider extending your lessons and ideas for improvement to your care team and health system. WHAT CAN I DO WITH WHAT I LEARNED? Step 1. Identify improvement opportunities. One way to translate these indepth case reviews into actual performance improvement is to find knowledge or skill gaps and try to address those gaps using concrete, actionable steps. You could brush up on the diagnostic criteria of a certain disease or the value of a certain test, or you could use available resources more to support your clinical reasoning. For instance, you could learn how to better leverage different features and functions of the EHR, access diagnostic testing guidelines, bookmark online knowledge resources, and identify experts in your practice or specialty and reach out to them for guidance or mentoring. Just as valuable, clinicians will often be able to identify their individual strengths and areas in which they are well calibrated. Step 2. Develop an action plan. Translate your insight into specific actions to sustain and integrate continuous learning. Some examples include: 1. Orally share lessons about a knowledge gap with three colleagues in order to internalize it. 2. If you had difficulties recognizing out-of-reference range values on uncommonly ordered lab tests, commit to looking up reference ranges and causes of false negatives/positives on tests you do not order frequently. 3. If you noticed absence of differential diagnoses on patients who had unexpected trajectories, commit to using knowledge resources to broaden your differentials at prespecified times (e.g., once per clinic day, once per shift, once per week on inpatient service). Step 3. Consider augmenting your case reviews with simulation. Simulation has been used effectively across medicine to help clinicians master a broad range of skills, from advanced procedural techniques to communicating more effectively with patients and beyond.21,22 While research has not yet quantified the effects of simulated cases on diagnostic calibration,23 simulated cases may be a useful complement to calibrate your clinical reasoning. In simulated cases, patients are presented in real-life clinical scenarios. Generally, clinical information is presented in parts, asking you as the learner to stop at various points to consider a differential diagnosis and next steps as if the patient were in front of you. As the case unfolds, you can compare your clinical reasoning with that of the authors, usually content experts, providing real-time feedback on your diagnostic process.24 This type of deliberate practice can help identify knowledge gaps and can weave the case presentation and diagnostic reasoning into your episodic memory the way an actual patient encounter often does.25 In addition, these cases are generally easily accessible, low cost, widely distributed, and amenable to remote practice.26 Following are several outlets that provide virtual simulated cases: New England Journal of Medicine - Interactive Case Series (NEJM Group) NEJM has featured these interactive cases since 2009.27 A subscription or access via a medical library is required. An archive of these cases can be found at: https://www.nejm.org/multimedia/interactive-medical-case.  11 New England Journal of Medicine - Case Records of the Massachusetts General Hospital (NEJM Group) A subscription or access via a medical library is required. An archive of these cases can be found at: https://www. nejm.org/medical-articles/case-records-of-the-massachusetts-general-hospital. Journal of General Internal Medicine - An Exercise in Clinical Reasoning A subscription or access via a medical library is required. An archive of these cases can be found at: https://www. springer.com/journal/11606. Search for “exercise in clinical reasoning.” Journal of Hospital Medicine - Clinical Care Conundrums A subscription or access via a medical library is required. An archive of these cases can be found at: https://www. journalofhospitalmedicine.com/jhospmed/clinical-care-conundrums. MedEdPORTAL (Association of American Medical Colleges) Available free of charge at https://www.mededportal.org/. Search for “simulation case” and “simulated case.” Medscape Available free with registration at https://www.medscape.com. Look under the “CME & Education” menu. Human Diagnosis Project Link to download the app available free of charge at https://www.humandx.org/. The Clinical Problem Solvers podcast Available free of charge at clinicalproblemsolving.com or through podcast streaming services. WHAT IS USEFUL FOR THE PATIENT’S CARE TEAM TO LEARN? Often, clinicians will be able to distill important lessons that can benefit the entire care team. Some institutions have structured channels of communication so that the entire interdisciplinary care team regularly discusses their shared patients. They may conduct daily interdisciplinary rounds, team huddles, or regular safety meetings, which could all serve as venues for case discussion.28,29 In other settings, communication between these groups is unstructured and on the fly. Calibration exercises could serve as an impetus for teams to create a more formal system for interdisciplinary case review. Calibration exercises could also identify cases for existing teams that review quality and safety events for improvement opportunities. WHAT IS USEFUL FOR PEERS WHO ARE NOT ON MY CARE TEAM? Did you identify practical insights that may be of value to other clinicians in your practice specialty or in similar settings? Check with the point of contact in your organization who can advise about sharing what you learned in a way that meets the requirements of any applicable patient privacy and patient safety/quality improvement confidentiality protections. One venue for dissemination may be morbidity and mortality or other quality improvement and safety conferences where clinicians from multiple departments and groups (e.g., risk management, laboratory, radiology, and members of quality committees) can learn and act on your findings and experiences. Other venues may include clinic- and service-level meetings, journal clubs, and local professional society meetings.  12 WHAT ARE THE LESSONS FOR THE BIGGER SYSTEM? Systems thinking is an essential additional lens to analyze diagnostic events. If a system issue contributed to a missed opportunity or near-miss in your practice, it likely also affected other clinicians and patients. Instead of devising a workaround, consider initiating a process that might result in a solution to improve care across the system. For example, if you missed visualizing an important test result because of how the EHR displayed information, you can recommend a change to the default display settings to your medical informatics team and the point of contact in your organization for patient safety improvement activities. More information on this topic can be found at: https://psnet.ahrq.gov/primer/systems-approach.  13 Reflect on Diagnostic Performance and Adjust if Needed MAIN IDEA Every round of case reviews should lead to additional reflection and insight into your diagnostic reasoning and related outcomes, thereby increasing your diagnostic calibration in a continuous learning process. Over time, consider alternating the foci of your calibration exercises between new clinical scenarios and domains and those scenarios and domains you examined in the past. You may also want to meet with different peers with varying areas and levels of expertise to broaden the perspectives you encounter when completing these calibration exercises. More perspectives will increase the chances that you will obtain an accurate assessment of your diagnostic reasoning, thus improving your calibration. Consider incorporating additional techniques to improve your calibration that you did not consider or engage in during your initial iterations of the exercise. For example, incorporate open discussion of your diagnostic reasoning and resulting outcomes with others if you only assessed your performance alone. Acknowledge and discuss diagnostic uncertainty with your colleagues and your patients.30 Lastly, adopt a perspective of “humility rather than heroism with [your] diagnostic decision-making capabilities.”31  14 Putting It All Together Let’s review how to use Calibrate Dx by walking through the steps with one clinician who decided to try it. Meet our (fictional) colleague, Dr. Nguyen. Dr. Nguyen is an early career hospitalist who often feels pressed for time during a typical day. After one of his patients unexpectedly decompensated and was transferred to intensive care, he wanted to figure out if his diagnostic reasoning was affected at moments of high stress, especially for his sicker patients. Implementing the Diagnostic Calibration Learning and Improvement Cycle STEP DESCRIPTION HOW DR. NGUYEN APPROACHED THIS STEP Choose an area of practice for which you Dr. Nguyen decided that he would track some of his patients would like to be better calibrated. SPECIFY the who had a rapid response team called so he could go back and You will likely learn more from focusing calibration task. review his own diagnostic performance. on a specific area of practice than reviewing your cases at random. After working with the Quality Management Office on needed permissions and logistics for himself and the peer he wanted to consult, he made a separate list in the electronic health record Select a small sample of your cases, and added patients when a rapid response team was called. EVALUATE review them for learning opportunities, Dr. Nguyen reviewed the cases from his list using the Revised diagnostic performance. and seek further feedback from a Safer Dx Instrument (Appendix B). After completing his case colleague. reviews, he summarized the process using the Diagnostic Calibration Debrief tool (Appendix A) and discussed what he learned with his colleague. See Dr. Nguyen’s completed case review tools in Appendix D. Dr. Nguyen thought that the differential diagnosis of new Identify improvement strategies for onset hypotension in hospitalized patients was an important yourself (and, when appropriate, your PLAN AND APPLY teaching point. He arranged to end a routine staff meeting 15 team and your system), and begin to improvement strategies. minutes early in order to share his findings with a small group of take appropriate action. Repeat the hospitalists. He also set a calendar reminder to repeat the case previous steps at regular intervals. review process in 6 months. Dr. Nguyen identified additional areas of interest based on this Reflect on this calibration exercise case to include differential diagnoses for high-risk changes in REFLECT on over time, evaluate additional areas hospitalized patients. He reviewed the recent literature on the diagnostic performance of interest, and make adjustments as differential diagnosis for hospitalized patients with new onset and adjust if needed. needed. dyspnea, new onset fever, and new onset delirium. Over time, he began to incorporate these strategies into his teaching.  15 References 1. Gawande A. Better: A Surgeon’s Notes on Performance. New York: Metropolitan; 2007. 2. Meyer AND, Singh H. The path to diagnostic excellence includes feedback to calibrate how clinicians think. JAMA. 2019 Feb;321(8):737-38. doi: 10.1001/jama.2019.0113. https://pubmed.ncbi.nlm.nih.gov/30735239/. Accessed August 16, 2022. 3. Singh H, Meyer AND, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving U.S. adult populations. BMJ Qual Saf. 2014 Sep;23(9):727-31. doi: 10.1136/ bmjqs-2013-002627. Epub 2014 Apr 17. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4145460/. Accessed August 16, 2022. 4. Singh H, Graber ML. Improving diagnosis in health care--the next imperative for patient safety. N Engl J Med. 2015 Dec;373(26):2493-95. doi: 10.1056/NEJMp1512241. Epub 2015 Nov 11. https://pubmed.ncbi.nlm.nih. gov/26559457/. Accessed August 16, 2022. 5. Meyer AN, Payne VL, Meeks DW, Rao R, Singh H. Physicians’ diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA Intern Med. 2013 Nov;173(21):1952-58. doi: 10.1001/jamainternmed.2013.10081. https://pubmed.ncbi.nlm.nih.gov/23979070/. Accessed August 16, 2022. 6. Omron R, Kotwal S, Garibaldi BT, Newman-Toker DE. The diagnostic performance feedback “calibration gap”: why clinical experience alone is not enough to prevent serious diagnostic errors. AEM Educ Train. 2018 Oct;2(4):339-42. doi: 10.1002/aet2.10119. Epub 2018 Sep 17. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6194049/. Accessed August 16, 2022. 7. Dhaliwal G. Annals for hospitalists inpatient notes - diagnostic excellence starts with an incessant watch. Ann Intern Med. 2017 Oct;167(8):H02-03. doi: 10.7326/M17-2447. https://pubmed.ncbi.nlm.nih.gov/29049776/. Accessed August 16, 2022. 8. Croskerry P. The feedback sanction. Acad Emerg Med. 2000 Nov;7(11):1232-38. doi: 10.1111/j.1553-2712.2000. tb00468.x. https://pubmed.ncbi.nlm.nih.gov/11073471/. Accessed August 16, 2022. 9. Fernandez Branson C, Williams M, Chan TM, Graber ML, Lane DP, Grieser S, Landis-Lewis Z, Cooke J, Upadhyay DK, Mondoux S, Singh H, Zwaan L, Friedman C, Olson APJ. Improving diagnostic performance through feedback: the Diagnosis Learning Cycle. BMJ Qual Saf. 2021 Dec;30(12):1002-9. doi: 10.1136/bmjqs-2020-012456. Epub 2021 Aug 20. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8606468/. Accessed August 16, 2022. 10. Cifra CL, Sittig DF, Singh H. Bridging the feedback gap: a sociotechnical approach to informing clinicians of patients’ subsequent clinical course and outcomes. 2021 Jul:30(7):591-97. doi: 10.1136/bmjqs-2020-012464. Epub 2021 May 6. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8237185/. Accessed August 16, 2022. 11. Ambrose SA, Bridges MW, DiPietro M, Lovett MC, Norman MK, Mayer RE. How Learning Works: Seven Research-Based Principles for Smart Teaching. San Francisco: Jossey-Bass; 2010. 12. Centers for Disease Control and Prevention. Health Equity Guiding Principles for Inclusive Communication. https://www.cdc.gov/healthcommunication/Health_Equity.html. Accessed August 16, 2022.  16 13. Singh H, Sittig DF. Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework. BMJ Qual Saf. 2015 Feb;24(2):103-10. doi: 10.1136/bmjqs-2014-003675. https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC4316850/. Accessed August 16, 2022. 14. National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press; 2015. 15. Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001. https://nap.nationalacademies.org/ catalog/10027/crossing-the-quality-chasm-a-new-health-system-for-the. Accessed August 16, 2022. 16. Croskerry P. 50 Cognitive and Affective Biases in Medicine. https://sjrhem.ca/wp-content/uploads/2015/11/ CriticaThinking-Listof50-biases.pdf. Accessed August 16, 2022. 17. Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019 Nov;6(4):315-23. doi: 10.1515/dx-2019-0012. https:// pubmed.ncbi.nlm.nih.gov/31287795/. Accessed August 16, 2022. 18. Reilly JB, Myers JS, Salvador D, Trowbridge RL. Use of a novel, modified fishbone diagram to analyze diagnostic errors. Diagnosis. 2014 Jun;1(2):167-71. doi: 10.1515/dx-2013-0040. https://pubmed.ncbi.nlm.nih.gov/29539996/. Accessed August 16, 2022. 19. Harada T, Miyagami T, Watari T, Kawahigashi T, Harada Y, Shikino K, Shimizu T. Barriers to diagnostic error reduction in Japan. Diagnosis (Berl). 2021 Jun. doi: 10.1515/dx-2021-0055. Epub ahead of print. https://pubmed.ncbi.nlm. nih.gov/34187115/. Accessed August 16, 2022. 20. Edwards JJ, Wexner S, Nichols A. Debriefing for Clinical Learning. PSNet Primers. Agency for Healthcare Research and Quality; November 2021. https://psnet.ahrq.gov/primer/debriefing-clinical-learning. 21. Kononowicz AA, Woodham LA, Edelbring S, Stathakarou N, Davies D, Saxena N, Tudor Car L, Carlstedt-Duke J, Car J, Zary N. Virtual patient simulations in health professions education: systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019 Jul;21(7):e14676. doi: 10.2196/14676. https://www.jmir. org/2019/7/e14676/. Accessed August 16, 2022. 22. Aggarwal R, Mytton OT, Derbrew M, Hananel D, Heydenburg M, Issenberg B, MacAulay C, Mancini ME, Morimoto T, Soper N, Ziv A, Reznick R. Training and simulation for patient safety. Qual Saf Health Care. 2010 Aug;19 Suppl 2:i34-43. doi: 10.1136/qshc.2009.038562. https://pubmed.ncbi.nlm.nih.gov/20693215/. Accessed August 16, 2022. 23. Graber ML, Kissam S, Payne VL, Meyer AND, Sorensen A, Lenfestey N, Tant E, Henriksen K, Labresh K, Singh H. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf. 2012 Jul;21(7):535-57. doi: 10.1136/bmjqs-2011-000149. Epub 2012 Apr 27. https://pubmed.ncbi.nlm.nih.gov/22543420/. Accessed August 16, 2022. 24. Motola I, Devine LA, Chung HS, Sullivan JE, Issenberg SB. Simulation in healthcare education: a best evidence practical guide. AMEE Guide No. 82. Med Teach. 2013 Oct;35(10):e1511-30. doi: 10.3109/0142159X.2013.818632. Epub 2013 Aug 13. https://pubmed.ncbi.nlm.nih.gov/23941678/. Accessed August 16, 2022. 25. Rajkomar A, Dhaliwal G. Improving diagnostic reasoning to improve patient safety. The Perm J. 2011 Summer;15(3):68- 73. doi: 10.7812/tpp/11-098. Epub 2011 Summer. https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC3200103/. Accessed August 16, 2022.  17 26. Plackett R, Kassianos AP, Kambouri M, Kay N, Mylan S, Hopwood J, Schartau P, Gray S, Timmis J, Bennett S, Valerio C, Rodrigues V, Player E, Hamilton W, Raine R, Duffy S, Sheringham J. Online patient simulation training to improve clinical reasoning: a feasibility randomised controlled trial. BMC Med Edu. 2020;20(1):245. doi: 10.1186/s12909- 020-02168-4. Epub 2020 Jul 31; https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7395338/. Accessed August 16, 2022. 27. New England Journal of Medicine. Interactive Medical Case. https://www.nejm.org/multimedia/interactive-medical- case. Accessed August 16, 2022. 28. Curley C, McEachern JE, Speroff T. A firm trial of interdisciplinary rounds on the inpatient medical wards: an intervention designed using continuous quality improvement. Med Care. 1998 Aug;36(8 Suppl):AS4-12. doi: 10.1097/00005650-199808001-00002. https://pubmed.ncbi.nlm.nih.gov/9708578/. Accessed August 16, 2022. 29. Pimentel CB, Snow AL, Carnes SL, Shah NR, Loup JR, Vallejo-Luces TM, Madrigal C, Hartmann CW. Huddles and their effectiveness at the frontlines of clinical care: a scoping review. J Gen Intern Med. 2021 Sept;36(9):2772-83. doi: 10.1007/s11606-021-06632-9. Epub 2021 Feb 8. https://pubmed.ncbi.nlm.nih.gov/33559062/. Accessed August 16, 2022. 30. Santhosh L, Chou CL, Connor DM. Diagnostic uncertainty: from education to communication. Diagnosis (Berl). 2019 Jun;6(2):121-26. doi: 10.1515/dx-2018-0088. https://pubmed.ncbi.nlm.nih.gov/30851157/. Accessed August 16, 2022. 31. Singh H, Zwaan L. Web Exclusives. Annals for hospitalists inpatient notes - reducing diagnostic error - a new horizon of opportunities for hospital medicine. Ann Intern Med. 2016 Oct;165(8):HO2-O4. doi: 10.7326/M16-2042. https:// pubmed.ncbi.nlm.nih.gov/27750328/. Accessed August 16, 2022.  18 List of Appendixes Appendix A. Diagnostic Calibration Debrief Tool Appendix B. Revised Safer Dx Instrument Appendix C. Modified Fishbone Diagram Appendixes D-G. Each of the following case examples includes a completed debriefing tool that summarizes the case review process, followed by a sample completed case review tool (Revised Safer Dx Instrument or Modified Fishbone Diagram) for one of the patient records that were reviewed in the process. Appendix D. Review of Rapid Responses Appendix E. Review of Pulmonary Emboli Diagnoses Appendix F. Review of Colorectal Cancer Diagnoses Appendix G. Review of Children Presenting With Vomiting  19 Appendix A. Diagnostic Calibration Debrief Tool (1 of 2) This tool can help you plan and summarize your efforts to calibrate your diagnostic performance and includes space for your self-assessment, peer feedback, and ideas for improvement. Do NOT record any patient health information (PHI), provider-specific information, or any other identifying information (e.g., dates) in this document. Section 1. Case Review Plan FOCUS OF CALIBRATION (Examples: diagnosis-specific situations, undifferentiated presentations, unexpected trajectories, diagnostic test interpretation, high- risk situations, your organization’s priorities, high-risk patient populations) PROCESS(ES) EVALUATED OUTCOME(S) EVALUATED Patient-provider Consultations and referrals Effectiveness Safety interactions Patient factors Timeliness Patient centeredness Test performance and Other Efficiency Equity interpretation Followup and tracking DATA SOURCE(S) OR SELECTION CRITERIA FOR RECORDS REVIEWED: (Examples: personal “remind-me” list, EHR query) CASE REVIEW TOOL USED: Revised Safer Dx Fishbone Diagram Other No specific tool used CALIBRATION QUESTION(S) CONSIDERED (SEE TABLE 3 FOR EXAMPLES): 1. 2. 3. Continued on next page  20 Diagnostic Calibration Debrief Tool (2 of 2) Section 2. Reflections and Assessment WRITE A SUMMARY ASSESSMENT AFTER REVIEWING YOUR CASES. CONSIDER THE FOLLOWING QUESTIONS AS YOU REFLECT ON YOUR DIAGNOSTIC PERFORMANCE: What did you learn from your case reviews? What, if anything, surprised you? How did you manage uncertainty in the diagnostic process? What went well? What will you repeat in similar cases in the future? What will you do differently in similar cases in the future? TAKE-HOME MESSAGE: SELF-REFLECTION AFTER DISCUSSING ASSESSMENT WITH PEER: NEXT STEPS: (e.g., plans for sharing, discussion, new initiatives)  21 Appendix B. Revised Safer Dx Instrument 1 2 3 4 5 6 7 Strongly Disagree Neutral Strongly Agree ITEM SCORE 1 The documented history was suggestive of an alternate diagnosis, which was not considered in the diagnostic process. 2 The documented physical exam was suggestive of an alternate diagnosis, which was not considered in the diagnostic process. 3 Data gathering through history, physical exam, and review of prior documentation (including prior laboratory, radiology, pathology, or other results) was incomplete, given the patient’s medical history and clinical presentation. 4 Alarm symptoms or “red flags” (i.e., features in the clinical presentation that are considered to predict serious disease) were not acted upon. 5 The diagnostic process was affected by incomplete or incorrect clinical information given to the care team by the patient or their primary caregiver. 6 The clinical information (i.e., history, physical exam, or diagnostic data) should have prompted additional diagnostic evaluation through tests or consults. 7 The diagnostic reasoning was not appropriate, given the patient’s medical history and clinical presentation. 8 Diagnostic data (laboratory, radiology, pathology, or other results) available or documented were misinterpreted in relation to the subsequent final diagnosis. 9 There was missed follow-up of available or documented diagnostic data (laboratory, radiology, pathology, or other results) in relation to the subsequent final diagnosis. 10 The differential diagnosis was not documented OR the documented differential diagnosis did not include the subsequent final diagnosis. 11 The final diagnosis was not an evolution of the care team’s initial presumed diagnosis (or working diagnosis). 12 The clinical presentation at the initial or subsequent presentation was mostly typical of the final diagnosis. 13 In conclusion, based on all the above questions, the episode of care under review has a missed opportunity to make a correct and timely diagnosis. Reprinted with permission from Singh et al. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-323.  22 Appendix B, cont’d How To Review a Case for Learning Opportunities Using the Revised Safer Dx Instrument Important: Before analyzing cases, reviewers should read the original manuscript that describes the development and use of the Revised Safer Dx Instrument, which is freely available: Singh H, Khanna A, Spitzmueller C, Meyer A. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-23. doi:10.1515/dx-2019-0012. WHAT YOU WILL NEED TO BEGIN: Approval to access medical records and patient identifiers for conducting this improvement activity Revised Safer Dx Instrument Additional case review tools (optional) ENSURE THAT YOU AND ANY OTHER REVIEWERS HAVE A SHARED UNDERSTANDING OF 1 DIAGNOSTIC ERROR Keep the fundamental question in mind: could something different have been done to make the correct diagnosis earlier? Make your judgments about clinicians’ decision making and diagnostic reasoning based on the information they had available at the time. Look for missed opportunities not only by clinicians but also by the care team, system, and patients. IDENTIFY THE EPISODE OF CARE TO EVALUATE 2 Usually involves all the care a patient received over a given period of time for a specific health problem they present with. Can span multiple encounters, including inpatient, emergency, and outpatient visits, or focus on a sole encounter such as a hospitalization. REVIEW THE CHART WITH A FOCUS ON DIAGNOSTIC PROCESS RATHER THAN THE 3 ULTIMATE OUTCOME Start by evaluating the clinical encounter (history, exam, tests ordered), as well as the initial presumed diagnosis or working differential diagnosis. Read through the chart to understand how the diagnostic processes and reasoning evolved rather than focusing on the ultimate accuracy of the diagnosis or any potential adverse outcome. Also look at progress notes, test results, referrals, consultant notes, and other documents that informed the diagnosis. Use current literature or guidelines to evaluate the diagnostic process.  23 Appendix B, cont’d How To Review a Case for Learning Opportunities Using the Revised Safer Dx Instrument ANSWER THE PROMPTS IN THE REVISED SAFER DX INSTRUMENT TO MAKE A 4 DETERMINATION ABOUT MISSED OPPORTUNITIES Prompts 1-12 ask you to evaluate the diagnostic processes at various stages such as history taking, physical exam, diagnostic testing, consulting, and clinical reasoning. The higher you score each prompt, the more likely you think there was a missed opportunity for diagnosis at this stage of the process. Prompt 13 asks you to look at the case as a whole and come to a final judgment as to whether there was a missed opportunity for diagnosis. Do not try to add up the numbers of each question to make any type of overall score. The questions are only to help you think through each item so you can make an overall assessment at the end with prompt 13. Write a few sentences to add context and explain your reasoning for your answer to prompt 13.  24 Appendix C. Modified Fishbone Diagram The modified fishbone has been widely adopted in healthcare settings to help patient safety experts understand the complex interplay of factors that contribute to a diagnostic error. The categories at the top of the diagram represent contributing factors related to cognition, while system-level factors are listed at the bottom. Although the facts of each case will vary, the goal in dissecting a case is to identify and categorize all the contributing factors you can find. This process can help clarify the specific causes of the error and guide quality improvement activities. An example of a completed fishbone diagram can be found in Appendix E. For more information about using the modified fishbone diagram, refer to the open-access manuscript below: Reilly JB, Myers JS, Salvador D, Trowbridge RL. Use of a novel, modified fishbone diagram to analyze diagnostic errors. Diagnosis 2014;1:2, 167-171. https://doi.org/10.1515/dx-2013-0040.  25 Appendix D. Review of Rapid Responses  26 Appendix D, cont’d Review of Rapid Responses  27 Appendix D, cont’d Review of Rapid Responses Note: This rapid response was for hypotension in a patient already being treated for sepsis. The patient’s decompensation was a progression of his known sepsis, so I do not think there was a missed diagnostic opportunity.  28 Appendix E. Review of Pulmonary Emboli Diagnoses  29 Appendix E, cont’d Review of Pulmonary Emboli Diagnoses  30 Appendix E, cont’d Review of Pulmonary Emboli Diagnoses This fishbone diagram outlines factors contributing to the case of delayed diagnosis of PE described in the debrief tool above.  31 Appendix F. Review of Colorectal Cancer Diagnoses  32 Appendix F, cont’d Review of Colorectal Cancer Diagnoses  33 Appendix F, cont’d Review of Colorectal Cancer Diagnoses This was a missed opportunity to diagnose colorectal cancer. Initial symptoms were thought to be related to constipation and hemorrhoids. Subsequent findings of weight loss and anemia prompted a workup, but it was delayed.  34 Appendix G. Review of Children Presenting With Vomiting  35 Appendix G, cont’d Review of Children Presenting With Vomiting  36 Appendix G, cont’d Review of Children Presenting With Vomiting My diagnosis was delayed in part because my history taking was hindered by a language barrier. I initially didn’t document a wide differential because I have seen so much gastroenteritis in clinic, especially lately. I think I did a good job of re- evaluating my differential and workup when symptoms persisted. AHRQ Pub. No. 22(23)-0047-2-EF October 2022