Chapter 2: Steps In The EBP Process PDF

CHAPTER 2 ...

CHAPTER 2  Steps in the EBP Process Step 1: Question Formulation Step 2: Evidence Search s Some Useful Web Sites s Search Terms s An Internet Search Using Google Scholar and PsycINFO s A Time-Saving Tip Step 3: Critically Appraising Studies and Reviews Step 4: Selecting and Implementing the Intervention s Importance of Practice Context s The Importance of Practice Context: A Policy Example Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. s How Many Studies Are Needed? s Client Informed Consent Step 5: Monitor Client Progress Feasibility Constraints Key Chapter Concepts Review Exercises Additional Readings A s discussed in Chapter 1, various authors have recommended a number of steps in the evidence-based practice (EBP) process. Although not all authors agree on the exact number of steps and how to label them, they generally agree that the number is around five or six. More important, there is wide agreement as to what is to be done overall, regardless of the number of steps used to order them. In this chapter, five main 28 Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 28 20/06/12 10:03 AM  Steps in the EBP Process 29 steps in the EBP process are discussed in more detail. The chapter ends with a discus- sion of various daunting feasibility constraints that you are likely to encounter in try- ing to implement these EBP steps in the real world of practice. STEP 1: QUESTION FORMULATION Before you start searching for evidence, you need to know what question you are trying to answer. Chapter 1 described six common types of EBP questions. One of the most common of the six questions pertains to ascertaining which interventions, programs, or policies have the best evidence supporting their effectiveness. Suppose, for example, that you are planning to establish a residential treatment facility for physically or sexually abused girls who have emotional or behavioral prob- lems, and you need to decide what treatment modalities to employ. Your EBP ques- tion might inquire as to which treatment modalities have the best evidence supporting their effectiveness with girls who share the projected characteristics of your clients. Alternatively, you might have reason in advance to suspect that one or more particu- lar modalities will be most effective based on what you have read or heard about it from a theoretical standpoint. For example, colleagues involved in a similar program elsewhere might rave about the great success they’ve experienced using eye movement desensitization and reprocessing (EMDR). Colleagues in another program might claim to have had little success with EMDR and much more success with exposure therapy. Perhaps you’ve read clinical books on both modalities, and both look equally promis- ing to you from a clinical standpoint. Consequently, instead of asking a broader ques- tion about the effectiveness of the gamut of possible treatment modalities, you might narrow your search to the question of whether EMDR or exposure therapy is more Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. effective with the types of clients you plan to treat. A tool that you might choose to use to construct your question about effectiveness or about what assessment tool to use is the PICO framework. PICO questions include four parts, which are represented by the four letters in the acronym, as illustrated in Table 2.1. Writing a question using this PICO framework can help you clearly articu- late your question and help you to develop search terms that capture each of these important elements. Before you begin to search the evidence, it can be helpful to lay out your question— whether you are using a PICO framework or not—so that you can think about the key search terms that you could use before you are in the thick of the search. Especially helpful is to think about synonyms that represent the same key concepts in your ques- tion. For example, different studies might refer to children aged 12 to 16 as children, youth, or young adolescents. If you don’t think about these synonyms, you may miss studies that just use different language to mean the exact same thing. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 29 20/06/12 10:03 AM  30 Overview of Evidence-Based Practice Table 2.1 The PICO Framework P Patient This is the part of the question where you would specify your client, the problem (or client), or need that you are addressing through services, and the key characteristics that population, you think might have an important impact on the intervention or assessment or problem tool that you choose. For example, some interventions have demonstrated different effects across different racial or ethnic populations, and some assessment instruments have not been developed to be used for clients with varied languages and cultures. This is important information to consider as you search for research and you consider the relevancy of the findings for your client. I Intervention For example, you may be trying decide between using two different parent C Comparison training interventions—or you may be thinking about adding a new depression screening instrument versus using clinical interviews to identify clients who need depression treatment. Choosing an intervention and a comparison is a good reminder that EBP questions involve making a practice decision; therefore, there should be at least two different options between which you will be deciding. The question should have real implications for directing what you actually do in practice. O Outcome This is what you’re hoping to see change as a result of your intervention or assessment efforts. You may be hoping to reduce child behavior problems among your clients, or more accurately detect moderate to severe depression. STEP 2: EVIDENCE SEARCH There are various ways to search for evidence related to your practice question. You could, for example, conduct an exhaustive literature search in a scholarly fashion. Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. Unless you are currently a student, however, your busy practice demands probably don’t leave enough time for that. Moreover, you probably lack access to some of the resources that students and academicians have, such as a university library (not to mention significant time to spend there) or a subscription to expensive profes- sional literature databases. Fortunately, if you have access to a more popular search engine such as Google, Yahoo!, or Bing, you might find some of the evidence you need there. For example, if you enter the search term exposure therapy or EMDR, Google, Yahoo!, or Bing will list a large number of links to Web sites on those treat- ment modalities. Most of the sites, however, will not enable you to read the original research stud- ies yourself. Instead, they’ll present summaries of the research and perhaps offer EBP guidelines. In Chapter 8 we will describe a rigorous approach to searching and synthe- sizing research called a systematic review. In that chapter you’ll learn what attributes to look for in a research review to give you a sense of the quality of the review. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 30 20/06/12 10:03 AM  Steps in the EBP Process 31 You should be cautious and exercise some healthy skepticism when encounter- ing Web sites that tout particular treatment approaches or other tools that you might use in your practice. Some sites, for example, might have a vested interest in promot- ing or debunking a particular treatment modality you are investigating. Table 2.2 lists 5 of the first 10 links that came up when I entered the search term EMDR in Google. The first two of the entries provide a brief, unbiased summary of the evidence sup- porting the effectiveness of EMDR as well as some of the questions and controver- sies about that evidence. The third and fourth entries are Web sites for organizations that promote EMDR. If you go to the third and fourth sites you will find information about the effectiveness of EMDR (including mention of some prestigious organizations that have designated it an effective treatment for posttraumatic stress), but you will not find any mention of the questions and controversies about its effectiveness. The fifth site provides an overview of those controversies and questions. If you go to any of the last three links, you will see a slanted discussion that attempts to either support or refute EMDR’s effectiveness. (I say this not to imply that I disagree with the points made in that discussion. In fact, I happen to believe that EMDR is effective under certain circumstances. It’s just that these discussions are not written in as unbiased a manner as are the summaries at the first two sites listed in Table 2.2.) Table 2.2 Web Search Example Using the Term EMDR at Google.com Eye movement desensitization and reprocessing - Wikipedia, the... en.wikipedia.org/.../Eye_movement_desensitization_and_reprocessin... Eye movement desensitization and reprocessing (EMDR) is a form of psychotherapy that was developed by Francine Shapiro to resolve the development of... Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. EMDR: Taking a Closer Look: Scientific American www.scientificamerican.com/article.cfm?id⫽emdr-taking-a-closer... Jan 3, 2008 – Can moving your eyes back and forth help to ease anxiety? EMDR Institute, Inc. www.emdr.com/ Offers therapy. Provides a history of the approach, including a discussion of the Adaptive Processing Model. Also includes references and contact information. EMDR International Association www.emdria.org/ The EMDR International Association (EMDRIA) is a professional association... eye movement desensitization and reprocessing (EMDR) - The... www.skepdic.com/emdr.html May 16, 2011 – Discusses the usefulness of eye movement in treatments. Focuses on studies and research data. Note: Search performed using Google on October 14, 2011. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 31 20/06/12 10:03 AM  32 Overview of Evidence-Based Practice Some Useful Web Sites You should look for Web sites that provide objective reviews. Two highly regarded sources of rigorous, objective reviews can be found at the Web sites of the Cochrane Collaboration and the Campbell Collaboration. Both of these sibling collabora- tions recruit groups of experts to conduct each review. The reviews provided by the Cochrane Collaboration focus on health care interventions and can be accessed at www.cochrane.org. In addition to its reviews, that site provides links to critical apprais- als of the reviews, bibliographies of studies, and other information, including informa- tion to help readers appraise the quality of their review system. The Campbell Collaboration reviews focus on social welfare, education, and crim- inal justice. You can access its Web site at www.campbellcollaboration.org. Even though the Cochrane and Campbell Collaborations are maintained separately, they do include overlapping areas of research. For example, reviews of mental health-relevant research can be found in both libraries of systematic reviews. Another other highly regarded source is the American Psychological Association’s Web site (www.apa.org/divisions/div12/rev_est/) on empirically supported treatments. Government sites can be another good option. One such site, for example, is the National Center for Post-Traumatic Stress (www.ncptsd.va.gov/publications/cq/v5/n4/ keane.html). Rather than rely exclusively on reviews, which as I have noted can be risky, you can review individual studies yourself. One way to do that is by going to the National Institutes of Health (NIH) government Web site (www.nlm.nih.gov) to get free access to a professional literature database called MedLine, which is provided by the National Library of Medicine and includes journal citations and abstracts. In response Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. to increased calls for public access to taxpayer-supported research, NIH implemented a new public access policy in 2008 that required that any published results of NIH- funded research be submitted to a digital archive and made accessible to the public no later than 12 months after publication. This digital archive is called PubMed Central. The PubMed Web site (www.ncbi.nlm.nih.gov/pubmed) provides access to both MedLine citations and abstracts, as well as full text articles when available through PubMed Central. Another option is provided by Google and is called Google Scholar. You can access it through Google by entering Google Scholar as your search term. This Google search option is designed to broadly identify scholarly literature and is helpful in narrowing down sources to journal articles, chapters, reports, and books. For example, when I entered the search term EMDR into Google Scholar, the first 10 sources were all peer- reviewed articles from journals such as the Journal of Consulting and Clinical Psychology and the Journal of Anxiety Disorders. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 32 20/06/12 10:03 AM  Steps in the EBP Process 33 Search Terms Whether you use Medline, Google Scholar, or some other professional literature data- base, the process for electronically retrieving individual studies is essentially the same. Typically, you begin by entering a search term connected to your practice question. For example, questions about what interventions are most effective in treating physically or sexually abused girls with posttraumatic stress disorder (PTSD)—or about the com- parative effectiveness of exposure therapy versus EMDR—might entail entering search terms like PTSD, exposure therapy, EMDR, sexual abuse, child abuse, and so on. On occasion you may find yourself searching for research on a topic with which you are relatively unfamiliar. In this case, you may struggle to confidently come up with search terms that exactly capture what you’re looking for. Many areas of practice might be described using jargon or terms that aren’t immediately obvious to a newcomer. It could save you some precious time to have a brief discussion about possible search terms with someone who is more familiar with the area of practice within which you are searching. Your search term at some sites can be rather long. For example, at Google Scholar I conducted a search using the terms: treatment outcome with sexually abused Native Alaskan girls with PTSD. (Actually, I didn’t need to use the word with in the search term, but I have kept it here to make the term more readable to you.) Alternatively, some databases will give you the option of coupling shorter search terms by using the connecting word and to more narrowly target the number of studies that will come up. For example, if you just enter the search term PTSD, you’ll get an overwhelming list of bibliographic references covering all aspects of PTSD. However, if your search terms include both PTSD and treatment outcome, the list will be shorter and will be limited to references dealing with treatment outcome in PTSD. If you want to reduce further Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. the number of irrelevant references that you’ll need to wade through, you can expand your search term using the word and twice. For example, if your search term contains PTSD and treatment outcome and sexually abused girls, you’ll get a much shorter list. You can also limit the types of research designs that will come up. For example, you can enter the search term PTSD and treatment outcome and randomized experiments and sexually abused girls. If you narrow your search term too much, however, you might not get enough useful references. For example, if your search term is PTSD and treatment outcome and randomized experiments and sexually abused girls and Native Alaskans, you’ll probably find no references that meet all your specifications. (I tried this and found none, although some tangential references did come up.) Finding the right types of research for your particular question requires that you know something about which types of research are best used to answer which kinds of questions. If your search term is so narrow that you get too few (or perhaps no) useful references, you should restart your search with a broader term—one less and, perhaps. Another Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 33 20/06/12 10:03 AM  34 Overview of Evidence-Based Practice way to broaden your search is by using the connecting word or. For example, if you are searching for studies on the treatment of trauma symptoms among children exposed to domestic violence, and your search term is treatment outcome and children and domestic violence, you might get more references to children who witness domestic violence and not get some useful ones on children who are themselves abused by a perpetrating family member. To broaden your search to pick up more of the latter type of useful references, you could use the search term treatment outcome and children and domestic violence or child abuse. In some cases, you may want to add the term not to specifically limit your search. For example, you may be searching for studies related to adult attention deficit disorder, but find that you are overwhelmed with studies of children. Your search might include the term: attention deficit disorder not child. The search terms and, or, and not are called Boolean operators, and all search engines include these operators to guide searches. Although many electronic databases have elements in common, such as Boolean operators, each of them has their own quirks. For example, some search engines assume that the connecting word is and, while others assume that the connecting word is or when you do not specify one or the other between terms. Some search engines allow for a smaller number of search terms than others. In general, it’s well worth your time to look at the help section of the database that you are using to learn how to use each engine to its fullest potential and save yourself some time. The feature titled “An Internet Search Using Google Scholar and PsycINFO” illus- trates how you might have to play around with a variety of search terms to find what you need and feel confident that you haven’t missed something relevant and valu- able. There is no one simple way to conduct your search, so you might ask, “Is it really worth the time and effort?” Well, what if you were the client in great distress and hop- Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. ing that the practitioner treating you cared enough to make every effort to provide you with the treatment that had the best chance of success? What would your answer be? An Internet Search Using Google Scholar and PsycINFO A while back, in preparing to give a lecture on the EBP process, I decided to con- duct an Internet search pertaining to a child I once treated. I conducted this search before gaining more expertise on using Internet bibliographic databases, so my expe- rience might resemble what you encounter when you start using these resources. My EBP question was as follows: What interventions have the best empirical support for treating the trauma symptoms of a 6-year-old African American boy who witnessed domestic violence? Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 34 20/06/12 10:03 AM  Steps in the EBP Process 35 I conducted this search in several phases. Not wanting to miss any kinds of studies bearing on my question, I began by using Google Scholar and entered my entire EBP question as the search term. (Actually, Google Scholar provides an advanced search option that would have allowed me to just enter certain words like interventions and trauma and domestic violence. The screen for that option is displayed in Figure 2.1.) Ten links came up, but only four of them seemed like they might be even remotely relevant to my EBP question. One turned out to be an article on assess- ment and intervention with parents to stabilize children who have witnessed vio- lence. It did not report a specific evaluation of a particular intervention. Another reviewed in very broad terms mental health treatments for children who are refu- gees. A third reported a study of the perspectives of battered mothers on the impact of domestic violence on urban preschool children, but did not evaluate an interven- tion for the children. The fourth was a book chapter that argued that the gender of the perpetrator and the victim are related to aggression during childhood. So far, not so good! So I tried the search term: effective interventions with traumatized children (again using Google Scholar, I really didn’t need to include the word with in Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. Figure 2.1 Screen for Advanced Search Option in Google Scholar (Continued) Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 35 20/06/12 10:03 AM  36 Overview of Evidence-Based Practice my search term). I found hundreds of abstracts, but only eight addressed treatment effects. Not fully satisfied with the results of that search, I tried again with a different search term: effective interventions with children who witness domestic violence. Still using Google Scholar, again I found hundreds of abstracts, most of which did not address treatment effects, and only two that were not already found in the previous search. Next, I went to the PsycINFO professional literature database to which I have access through the university where I work. Many local libraries are now making PsycINFO and other related databases available at no cost to the public. These databases can be accessed remotely from your office or home free of charge. PsycINFO provides a drop-down menu in between the search terms that includes and/or/not. It also includes a methodology search filter where you can select the types of studies you’re looking for, such as an empirical study, treatment outcome study, and so on. You might want to enter the methodology search filter that best fits your research question first, and then work your way down to other types of research designs depending on what you find or don’t find. (Chapter 3 discusses what types of research designs tend to go best with different types of EBP questions.) Wanting to start with a very broad search before attempting to narrow it down, I used the keywords: treatment and traumatized children. Alternatively, I could have used filters to narrow things down regarding research designs or my client’s charac- teristics. When you conduct a search, it might be best to try it both ways. Then you can get a sense of what kinds of evidence you’ll find using either approach. In my broad search on PsycINFO, however, I found only four references that seemed to be relevant to my EBP question. Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. Consequently, I returned to Google Scholar to see what would happen if in addi- tion to the main search term, effective interventions with children who witness domes- tic violence, I added the term experimental design in the next box. In other words, I searched for links to sources that had all of the words effective interventions with children who witness domestic violence anywhere in the reference and with the exact phrase experimental design anywhere in the reference. This time, 153 references appeared. Thus, by adding the filter term experimental design, I came up with a more manageable list of articles. The vast majority of them, however, were still not ade- quately relevant to my EBP question. For example, the irrelevant ones addressed things like the prevention of violence among adult perpetrators, the treatment of battered women, dating violence among older children, and so on. Moreover, by stipulating that I was interested only in studies employing experimental designs, I was filtering out studies using other designs that might have been more relevant to my EBP question and the particular client characteristics in it. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 36 20/06/12 10:03 AM  Steps in the EBP Process 37 Finally, I repeated the last step, but this time entering the words battered women in the box labeled without the words. (See Figure 2.1 for a display of the window for the advanced search option in Google Scholar.) Doing so, however, cut the number of references to only 19, and none of them addressed empirical support for the effec- tiveness of interventions germane to my client. So, what lessons can be drawn from my various searches? One is that there is no pat answer—no panacea—for simplifying your Internet database search for studies relevant to your EBP question. Even if you use the and conjunction with filter terms, you might have to wade through many irrelevant references to find the ones that are pertinent to your EBP question. Moreover, by narrowing your search that way, you might miss some studies that are pertinent. Instead of expecting to find what you need easily with one set of search terms, you’ll probably have to play around with a variety of terms to be sure you aren’t missing something relevant and valu- able. Things can get pretty complicated very quickly as you try searching different combinations of terms, synonyms, and databases. Some databases track the search terms that you have used, while others do not. If the database you are using doesn’t automatically track your search terms, it’s likely worth the effort to record the search terms that you are trying. Because searching can be a bit of an art form, and you will likely try several combinations of words and phrases, it can quickly become easy to forget which terms you have tried. Keeping a sheet of paper nearby to jot down your search terms along the way can help you avoid repeating your efforts and wondering, “Did I remember to try searching for the terms family violence, domes- tic violence, and interpersonal violence?” Also, you’ll probably have to wade through Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. many irrelevant references to be sure to find the real gems. When working with similar clients, you may not need to repeat this process each time. What you find the first time might apply again and again. This is especially true if many of your clients experience similar problems and needs. However, keep in mind that the evidence might change over time. Therefore, if several months or more elapse after your search, you might want to repeat it to see if any new studies have emerged supporting different interventions. Moreover, some newer studies might be more appli- cable to your client’s unique characteristics or your unique practice situation. A Time-Saving Tip When conducting your own search, you don’t have to read every study that you find. You can examine their titles and abstracts to ascertain which ones are worth reading. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 37 20/06/12 10:03 AM  38 Overview of Evidence-Based Practice For example, some years ago, when I conducted a review of the effectiveness of EMDR versus exposure therapy in treating PTSD, I encountered an abstract depicting a study that concluded that EMDR helps “bereaved individuals experience what they believe is actual spiritual contact with the deceased” (Botkin, 2000, p. 181). I could tell from the title of the study that it was not relevant to my review regarding PTSD. (But given its bizarre claim, I read it anyway!) You can decide which studies to read based on the relevance of the study to your practice question as well as any mention in the abstract of attributes that might tip you off about the quality of the study. For example, if one abstract tells you that the study offers a practitioner’s anecdotal account of providing exposure therapy to one client, and another abstract depicts a large, multisite experiment evaluating the effectiveness of exposure therapy with many clients, you might be more predisposed to read the lat- ter study. Much of this book is devoted to giving you the information and understand- ing you’ll need to appraise the quality of the various studies you’ll find. This brings us to the next step in the EBP process: critically appraising studies and reviews. STEP 3: CRITICALLY APPRAISING STUDIES AND REVIEWS As I’ve already intimated, the individual studies and reviews that you’ll find in your search might vary greatly in regard to their objectivity and rigor. The journal peer review process offers a level of quality assurance insofar as published articles in peer- reviewed journals have at least been exposed to some process of review and cri- tique. However, the rigor of the review process in journals can vary greatly. Some very strong research studies do not appear in journal articles, while some relatively Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. weak studies do get published. Some studies and reviews, whether in journal arti- cles or other sources, for example, will be conducted and reported by individuals or groups with vested interests. But reviews and studies can be flawed even when no vested interests are involved. Some objective investigators do the best they can with limited resources to overcome some practical obstacles that keep them from imple- menting their study in a more ideal manner. A while back, for example, I conducted an experiment evaluating the effectiveness of EMDR in a child guidance center (Rubin et al., 2001). I had no funding for the study and conducted it simply because—as a professor—I am expected to do research and I was quite curious about whether EMDR was really as effective with children as its proponents were touting it to be. The admin- istrative and clinical leaders in the center projected that in a year’s time over 100 cli- ents would participate in my study. They were wrong. It took 3 years for them to refer 39 clients to my study. Some flaws are egregious and fatal. That is, they destroy the credibility of the study’s findings. To illustrate a fatally flawed fictional study, suppose Joe Schmo invents a new Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 38 20/06/12 10:03 AM  Steps in the EBP Process 39 therapy for treating anxiety disorders. He calls it psyschmotherapy. If it is effective, he will be rich and famous. To evaluate its effectiveness, he uses his own clinical judg- ment to rate client anxiety levels—on a scale from 0 to 100—before and after he pro- vides psyschmotherapy to 10 of his clients. His average before rating is 95, indicating extremely high anxiety. His average after rating is 10, indicating extremely low anxi- ety. He concludes that psyschmotherapy is the most effective intervention available for treating anxiety disorders—a miracle cure, so to speak. You probably can easily recog- nize the egregious bias and utter lack of trustworthiness evident in Joe Schmo’s study. Other flaws are more acceptable. For example, suppose instead of using the fore- going evaluation approach, Joe Schmo proceeds as follows. He collaborates with a colleague who works in his mental health clinic and who specializes in cognitive– behavioral treatment for anxiety disorders. Joe provides psyschmotherapy to the first 10 new clients referred to him for treatment of anxiety disorders. His colleague pro- vides cognitive–behavioral treatment to the first 10 new clients referred to him with anxiety disorders. To measure outcome, a graduate student who does not know what the study is about is hired to interview clients briefly before and after treatment and ask them to rate their average daily anxiety level (from 0 to 100) during the previ- ous 7 days. Regardless of the findings, we can see that this study is more credible than the previous one. It has flaws, but its flaws are neither egregious nor fatal. Maybe, for example, there are some differences in the types of clients referred to the two thera- pists, making one group more likely to improve than the other. Maybe all the clients in both groups exaggerated the improvements in their anxiety levels because they wanted to believe the treatment helped them or wanted the study’s findings to please their therapist. Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. While these flaws may not be fatal, they are important. If you can find studies less flawed than this one, you’d probably want to put more stock in their findings. But if this study is the best one you can find, you might want to be guided by its findings. That is, it would offer somewhat credible—albeit quite tentative—evidence about the comparative effectiveness of the two treatment approaches. Lacking any better evi- dence, you might want—for the time being—to employ the seemingly more effective approach until better evidence supporting a different approach emerges or until you see for yourself that it is not helping your particular client(s). Unlike these fictional examples, it is not always so easy to differentiate between rea- sonable “limitations and fatal flaws; that is, to judge whether the problems are serious enough to jeopardize the results or should simply be interpreted with a modicum of caution” (Mullen & Streiner, 2004, p. 118). What you learn in the rest of this book, however, will help you make that differentiation and thus help you judge the degree of caution warranted in considering whether the conclusions of an individual study or a review of studies merit guiding your practice decisions. Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 39 20/06/12 10:03 AM  40 Overview of Evidence-Based Practice STEP 4: SELECTING AND IMPLEMENTING THE INTERVENTION As discussed earlier in this chapter, a common misinterpretation of EBP is that you should automatically select and implement the intervention that is supported by the best research evidence, regardless of your practice expertise, your knowledge of idiosyn- cratic client circumstances and preferences, and your own practice context. No matter how scientifically rigorous a study might be and no matter how dramatic its findings might be in supporting a particular intervention, there always will be some clients for whom the intervention is ineffective or inapplicable. When studies declare a particu- lar intervention a “success,” this is most often determined by group-level statistics. In other words, the group of clients who received the successful intervention had better outcomes, on average, than those who did not receive the intervention. This doesn’t give us much information about how any given individual might have responded to the intervention. In practice, we are interested in successfully treating each and every client, not just the average. Moreover, we often don’t know why some clients don’t benefit from our most effec- tive interventions. Suppose an innovative dropout prevention program is initiated in one high school, and 100 high-risk students participate in it. Suppose a comparable high school provides routine counseling services to a similar group of 100 high-risk students. Finally, suppose only 20 (20%) of the recipients of the innovative program drop out, as compared to 40 (40%) of the recipients of routine counseling. By cutting the dropout rate in half, the innovative program would be deemed very effective. Yet it failed to prevent 20 dropouts. Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. Importance of Practice Context When an intervention is considered to be more effective (or more evidence based) than alternative interventions, it simply means that it has the best evidence indicating that it has the best likelihood of being effective. That is a far cry from implying that it will be effective with every case or even most cases or in all practice situations. Furthermore, we may know of important differences between our clients or practice context and the clients or contexts that comprised the studies supplying the best evidence. Perhaps we are serving clients from ethnic groups that did not participate in the best studies. Perhaps our clients are younger or older than the studied clients. Perhaps we are manag- ing an agency in which our practitioners have less experience and training in the sup- ported intervention than was the case in the best studies. Consequently, they may not be as effective in providing the intervention. Maybe their inexperience, or perhaps their heavier caseloads, will make them completely ineffective providers of that intervention. An autobiographical story from my experience illustrates the importance of practice Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 40 20/06/12 10:03 AM  Steps in the EBP Process 41 context in the feature titled “The Importance of Practice Context: A Policy Example.” That section also illustrates the applicability of EBP to administrative policy decisions. In light of the influence of practice context, deciding which intervention to imple- ment involves a judgment call based in part on the best evidence; in part on your practice expertise; in part on your practice context; and in part on the idiosyncratic characteristics, values, and preferences of your clients. While you should not underes- timate the importance of your judgment and expertise in making the decision, neither should you interpret this flexibility as carte blanche to allow your practice predilec- tions to overrule the evidence. The fact that you are well trained in and enjoy pro- viding an intervention that solid research has shown to be ineffective or much less effective than some alternative is not a sufficient rationale to eschew alternative inter- ventions on the basis of your expertise. Likewise, you should not let your practice pref- erences influence your appraisal regarding which studies offer the best evidence. You should base that appraisal on what you learn from reading subsequent chapters in this book. Just as you would not want to be treated by a physician who mechanically prescribes treatments for you without first learning about your idiosyncratic health factors (an allergy to the most effective medication, for example), neither would you want to be treated by a physician who does a thorough diagnosis and takes a thorough health history and then provides a treatment solely based on his or her comfort level and experience in providing that treatment, ignorant of or dismissing research on its effectiveness solely on the basis of his or her own predilections. The Importance of Practice Context: A Policy Example Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. Several decades ago, assertive case management came to be seen as a panacea for helping severely mentally ill patients dumped from state hospitals into communities in the midst of the deinstitutionalization movement. Studies supporting the effective- ness of assertive case management typically were carried out in states and communi- ties that provided an adequate community-based service system for these patients. Likewise, ample funding enabled the case managers to have relatively low caseloads, sometimes less than 10 (Rubin, 1992). One study assigned only two cases at a time to their case managers and provided them with discretionary funds that they could use to purchase resources for their two clients (Bush, Langford, Rosen, & Gott, 1990). These were high-quality studies, and their results certainly supported the effectiveness of assertive case management when provided under the relatively ideal study conditions. I had recently moved from New York to Texas at the time that those stud- ies were emerging. My teaching and research in those days focused on the plight of (Continued) Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 41 20/06/12 10:03 AM  42 Overview of Evidence-Based Practice the deinstitutionalized mentally ill. Included in my focus was the promise of, as well as issues in, case management. My work brought me into contact with various case managers and mental health administrators in Texas. They pointed out some huge discrepancies between the conditions in Texas compared to the conditions under which case management was found to be effective in other (northern) states. Compared to other states, and especially to those states where the studies were con- ducted, public funding in Texas for mental health services was quite meager. Case managers in Texas were less able to link their clients to needed community services due to the shortage of such services. Moreover, the Texas case managers lamented their caseloads, which they reported to be well in excess of 100 at that time. One case manager claimed to have a caseload of about 250! To these case managers, the studies supporting the effectiveness of assertive case management elsewhere were actually causing harm in Texas. That is, those studies were being exploited by state politicians and bureaucrats as a way to justify cutting costlier direct services with the rationale that they are not needed because of the effectiveness of (supposedly cheaper) case management services. How Many Studies Are Needed? One of the thornier issues in making your intervention decision concerns the num- ber of strong studies needed to determine which intervention has the best evidence. For example, will 10 relatively weak, but not fatally flawed, studies with positive results Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. supporting Intervention A outweigh one very strong study with positive results supporting Intervention B? Will one strong study suggesting that Intervention C has moderate effects outweigh one or two relatively weak studies suggesting that Intervention D has power- ful effects? Although we lack an irrefutable answer to these questions, many EBP experts would argue that a study that is very strong from a scientific standpoint, such as one that has only a few trivial flaws, should outweigh a large number of weaker studies containing serious (albeit perhaps not fatal) flaws. Supporting this viewpoint is research that suggests that studies with relatively weaker methodological designs can overestimate the degree of effectiveness of interventions (e.g., Cuijpers, van Straten, Bohlmeijer, Anderson, & Hollon, 2010; Wykes, Steel, Everitt, & Tarrier, 2008). If you find that Intervention A is supported by one or two very strong studies and you find no studies that are equally strong from a scientific standpoint in supporting any alternative interventions, then your findings would provide ample grounds for considering Intervention A to have the best evidence. However, determining that Intervention A has the best evidence is not the end of the story. Future studies might refute the current ones or might show newer interventions to Rubin, Allen, et al. Practitioner's Guide to Using Research for Evidence-Based Practice, John Wiley & Sons, Incorporated, 2012. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/utoronto/detail.action?docID=977922. Created from utoronto on 2024-10-16 15:19:09. c02.indd 42 20/06/12 10:03 AM  Steps in the EBP Process 43 be more effective than Intervention A. Although Intervention A might have the best evidence for the time being, you should remember that EBP is an ongoing process. If you continue to provide Intervention A for the next 10 or more years, your decision to do so should involve occasionally repeating the EBP process and continuing to find it to have the best supportive evidence. Client Informed Consent There may be reasons why Intervention A—despite having the best evidence—is not the best choice for your client. As discussed, your client’s idiosyncratic characteristics or your practice context might contraindicate Intervention A and thus influence you to select an alternative intervention with the next best evidence base. And even if you conclude that Intervention A is the best choice for your client, you should inform the client about the evidence and involve the client in making decisions about which interventions to use. I am not suggesting that you overwhelm clients with lengthy, detailed descriptions of the evidence. You might just tell them that based on the research so far, Intervention A appears to have the best chance of helping them. Be sure to inform them of any undesirable side effects or discomfort they might experi- ence with that intervention, as well as the possibility that the treatment may not work for them. With this information, the client might not consent to the treatment, in which case you’ll need to consider an alternative intervention with the next best evi- dence base. A side benefit of engaging the client in making an informed decision is that doing so might improve the client’s commitment to the treatment process, which in turn might enhance the prospects for a successful treatment outcome. Recall from our discussion in Chapter 1 that some of the most important factors influencing ser- Copyright © 2012. John Wiley & Sons, Incorporated. All rights reserved. vice effectiveness are related to the quality of the client–practitioner relationship. STEP 5: MONITOR CLIENT PROGRESS Before you begin to provide the chosen intervention, you and the client should iden- tify some measurable treatment goals that can be monitored to see if the intervention is really helping the client. This phase is important for several reasons. One reason, as noted previously, is even our most effective interventions don’t help everybody. Your client may be one of the folks who doesn’t benefit from it. An