Critically Appraise the Applicability and Quality of an Intervention Research Study PDF

3 **Critically Appraise** **the Applicability and** **Quality of an Intervention** **Research Study** **C H A P T E R - A T- A - G L A N C E** *This chapter will help you understand the appraisal of the* *following:* n Applicability of research evidence n Quality of research evidence **32**...

3 **Critically Appraise** **the Applicability and** **Quality of an Intervention** **Research Study** **C H A P T E R - A T- A - G L A N C E** *This chapter will help you understand the appraisal of the* *following:* n Applicability of research evidence n Quality of research evidence **32** **P R E - T E S T** Read the clinical questions and abstract below1: **S e a rchabl e C l i n i c a l** **Q u e s t i o n :** "For a 6-year-old girl with hemiplegia resulting from cerebral palsy, is constraint-induced movement therapy efficacious for improving the functional use of her hemiparetic arm?" **Abstract** **Background and Purpose**: This single-blinded randomized controlled trial compared the efficacy of a reimbursable, outpatient, modified constraint-induced therapy (mCIT) protocol (half-hour therapy sessions occurring 3 days per week in which subjects used the more affected arm combined with less affected arm restriction 5 days per week for 5 hours; both of these regimens were administered during a 10-week period) with that of a time-matched exercise program for the more affected arm or a no-treatment control regimen. **Subjects**: Thirty-five subjects with chronic stroke participated in the study. **Methods**: The Action Research Arm Test (ARAT), Fugl-Meyer Assessment of Motor Recovery After Stroke (FM), and Motor Activity Log (MAL) were administered to the subjects. **Results**: After intervention, significant differences were observed on the ARAT and MAL Amount of Use and Quality of Movement scales, all in favor of the mCIT group. **Discussion and Conclusion**: The data affirm previous findings suggesting that this reimbursable, outpatient protocol increases more affected arm use and function. Magnitude of changes was consistent with those reported in more intense protocols, such as constraint-induced therapy. Page, S., Levine, P., Leonard, A., Szaflarski, J., Kissela, B. Modified Constraint-Induced Therapy in Chronic Stroke: Results of a Single-Blinded Randomized Controlled Trial *PHYS THER March 2008 88:333-340* *\ *n**Introduction: The Process** **of Study Appraisal** This chapter and Chapter 4 support your knowledge and skill development for the third step of evidence based practice (EBP): **Step 3:** Critically **appraise** the research evidence for applicability and quality Step 3 in the EBP process is study appraisal. The appraisal process is summarized in the four parts in Figure 3.1. The first two parts are detailed in this chapter including appraising the: A. *Applicability and* B. *Quality of* a research study. The third part, appraising study *results,* and fourth part, formulating a *clinical bottom line,* are detailed in Chapter 4. Highquality studies produce valid results. Chapters 3 and 4 focus on the appraisal of intervention studies. When results of an intervention study are valid, they can more confidently be applied to inform patient care. **Part A: Determining** **Applicability of an** **Intervention Study** Using Research Evidence to Inform Care for Your Patients You may have a specific patient in mind when you search the literature, or you may have a more general question about types of patients with shared problems and what interventions are most effective for these types of patients. In both situations, you will examine the research you locate to determine if it applies to your patient or group of patients with a similar problem. The search engine PubMed was used to locate a randomized clinical trial (RCT) for 17-year-old swimmer June Wilson (Case Study 1.1). This article addressed the specific clinical question regarding a type of treatment and its effects on pain: **S e a rchabl e C l i n i c a l Q u e s t i o n :** "For a 17-year-old swimmer with neck pain, is a combination of manual therapy and exercise effective for reducing pain and improving function?" ***Article from search:*** Walker MJ, Boyles RE, Young BA, et al. The effectiveness of manual physical therapy and exercise for mechanical neck pain a randomized clinical trial. *Spine.* 2008;33:2371-2378.2 The explanatory sections that follow are organized by questions to consider when appraising the applicability of an intervention study. (Questions 1 through 5 are in Table 3.3.) ***QUESTION 1: Is the study's purpose relevant to my*** ***clinical question?*** As a first step in the appraisal, you determine if this study could apply to June. The article may be an example of excellent research with valuable results, but June is a unique person with unique circumstances, and this article may or may not be applicable to her. The first step is to review the abstract from the Walker et al2 study (Fig. 3.2). Read the notes in the abstract, and determine if this study would apply to June Wilson (Case Study 1.1). ***QUESTION 2: Is the study population (sample) sufficiently*** ***similar to my patient to justify the expectation that my*** ***patient would respond similarly to the population?*** Questions 2 and 3 should be appraised together. ***QUESTION 3: Are the inclusion and exclusion criteria*** ***clearly defined, and would my patient qualify for the study?*** You want to know the characteristics of the people studied. Are they similar enough to June or just too different for the results **CHAPTER 3 Critically Appraise the Applicability and Quality of an Intervention Research Study 33** **P R E - T E S T ---cont'd** **1.** How will you determine if this study is applicable to your patient? **2.** State one major threat to the quality of this study. **A Applicability** Clinical Question Search **Appraise:** **Intervention Studies** Integrate Evaluate **Step** **2 Step** **1** **3 Step** **4 Step** **5 Step** Quality Results Clinical Bottom Line **B** **C** **D** **\ 34 SECTION I Finding and Appraising Evidence to Improve Patient Care** **Study Design**: Randomized clinical trial. **Objective**: To assess the effectiveness of manual physical therapy and exercise (MTE) for mechanical neck pain with or without unilateral upper extremity (UE) symptoms, as compared to a minimal intervention (MIN) approach. **Summary of Background Data**: Mounting evidence supports the use of manual therapy and exercise for mechanical neck pain, but no studies have directly assessed its effectiveness for UE symptoms. **Methods**: A total of 94 patients referred to 3 physical therapy clinics with a primary complaint of mechanical neck pain, with or without unilateral UE symptoms, were randomized to receive MTE or a MIN approach of advice, motion exercise, and subtherapeutic ultrasound. Primary outcomes were the neck disability index, cervical and UE pain visual analog scales (VAS), and patient-perceived global rating of change assessed at 3-, 6-, and 52-weeks. Secondary measures included treatment success rates and post-treatment healthcare utilization. **Results**: The MTE group demonstrated significantly larger reductions in shortand long-term neck disability index scores (mean 1-year difference \_5.1, 95% confidence intervals (CI) \_8.1, to \_2.1; *P* \_ 0.001) and short-term cervical VAS scores (mean 6-week difference \_14.2, 95% CI \_22.7 to \_5.6; *P* \_ 0.001) as compared to the MIN group. The MTE group also demonstrated significant within group reductions in short- and long-term UE VAS scores at all time periods (mean 1-year difference \_16.3, 95% CI \_23.1 to \_9.5; *P* \_ 0.000). At 1-year, patient perceived treatment success was reported by 62% (29 of 47) of the MTE group and 32% (15 of 47) of the MIN group (*P* \_ 0.004). **Conclusion**: An impairment-based MTE program resulted in clinically and statistically significant short- and long-term improvements in pain, disability, and patient perceived recovery in patients with mechanical neck pain when compared to a program comprising advice, a mobility exercise, and subtherapeutic ultrasound. **Key Words**: mechanical neck pain, cervical pain, radicular pain, radiculitis, manual therapy, manipulation, mobilization, exercise. *Spine* 2008;33:2371-2378. **Note 3.3** **The study identifies a** **treatment that** **resulted in significant** **reduction in pain** **which is one of June's** **goals.** **Note 3.2** **The outcome** **measures include** **June's goal of** **reducing pain. The** **neck disability index** **may measure some** **aspects that will** **apply to June's other** **goal of continuing** **competitive** **swimming.** **Note 3.1** **The number and** **characteristics of the** **subjects in a study** **will be described in** **the *Method* section of** **a RCT. There were** **94 subjects from 3** **physical therapy** **clinics and the** **subjects' primary** **complaint was** **mechanical neck pain** **with or without** **upper extremity pain.** **So far this study** **could apply to June.** **She has mechanical** **neck pain without** **upper extremity pain.** **FIGURE 3.2** Abstract from Walker MJ, Boyles RE, Young BA, et al. The effectiveness of manual physical therapy and exercise for mechanical neck pain: a randomized clinical trial. *Spine.* 2008;33:2371-2378; with permission. **A** **Determining Applicability of** **Intervention Study** **Determining Quality of** **Intervention Study** Interpreting **Results** of Intervention Study Summarizing the **Clinical Bottom** **Line** of Intervention Study **B** **C** **D** **1** Identify the need for information and develop a focused and searchable **clinical question**. Conduct a **search** to find the best possible research evidence to answer your question. **Critically appraise the research evidence for** **applicability and quality: Intervention Studies.** **Integrate** the critically appraised research evidence with clinical expertise and the patient's values and circumstances. **Evaluate** the effectiveness and efficacy of your efforts in Steps 1--4 and identify ways to improve them in the future. **Step** **2 Step** **3 Step** **4 Step** **5 Step** **FIGURE 3.1** Four parts of study appraisal. to be applicable to June? Recall that you will most likely not find a study of people who are exactly like June, but they may have characteristics that are close enough for the results to have relevance for June. The inclusion and exclusion criteria for subjects are typically listed in the Methods section of a research paper. These criteria affect the applicability of the study. A treatment may be thought to be most effective in a restricted sample of patients with very specific criteria. To determine if the treatment is effective, a study may limit the subjects to these specific inclusion criteria. For example, a treatment intended for patients with cervical pain but without radiculopathy would most likely exclude any patient with radiculopathy. This limits the applicability of the study, but it identifies the specific patient group that will benefit from the treatment. What other information might you want to know about the subjects in this study? June is a teenager, but the abstract (see Fig. 3.2) does not include the ages of the study subjects. Subject characteristics are typically listed in tables within articles. Table 3.1 (from the Walker et al2 study) lists the participant characteristics, including the average age and variation in age (standard deviation) of subjects in the study. Subjects were middle-aged and did not include a teenage range. In thinking about the applicability of this research to June, you would ask, "Are there specific characteristics of the teenage period that would make research with older adults less applicable to June?" The search in Chapter 2 also identified a case series article that might include subjects that are more applicable to June (Case Study 1.1) (Fig. 3.3). The abstract in Figure 3.3 includes the age of the subjects, but these subjects also are middle-aged. Remember, you will not find the perfect study, but more typically you will find studies that are close enough to be applicable at least to some degree for your patient. When determining the applicability of a study to your patients, consider the following questions: 1\. Is my patient in an age range that has special considerations, for example, children, teenagers, late elderly? What aspects of their age range are relevant? For example, children will respond differently in comparison to adolescents or elders to a strength-building program because the physiology of their muscles and their motivational characteristics are different. 2\. Does my patient have a social, ethnic, cultural, or religious background that might constrain a particular type of treatment? For example, your patient might have limited insurance coverage for physical therapy, which might constrain an extended treatment period. 3\. Are the treatments under study feasible in my clinical setting? Is special equipment or skill necessary to duplicate the treatment effectively? For example, to use the results from the Walker et al2 study, you would need manual therapy skills. ***QUESTION 4: Are the intervention and comparison/*** ***control groups receiving a realistic intervention?*** The title of the study indicates that the study purpose included treatment of mechanical neck pain, a symptom reported by June. The two treatments that were compared, manual therapy and exercise, are also common physical therapy interventions that you might consider including in June's plan of care. **CHAPTER 3 Critically Appraise the Applicability and Quality of an Intervention Research Study 35** TABLE 3.1 **Baseline Characteristics of** **94 Participants** **MANUAL THERAPY MINIMAL** **AND EXERCISE INTERVENTION** **VARIABLE (n = 47) (n = 47)** Age (yr) 48.8 (14.1) 46.2 (15.0) Female gender 31 (66) 32 (68) Symptom duration (d)\* 1082 (365) 521 (70) Medications use 31 (66) 31 (66) Range of motion (degrees) Flexion 45.0 (14.1) 46.2 (14.4) Extension 42.7 (14.3) 40.4 (13.8) Rotation 50.9 (14.6) 51.9 (15.0) Sidebending 32.9 (12.6) 31.9 (10.0) Headache symptoms 27 (57) 32 (68) Upper extremity 31 (66) 27 (57) symptoms Data are mean (SD) for continuous variables or No. (%) for categorical variables, unless otherwise stated. \*Data are mean (median). From Walker MJ, Boyles RE, Young BA, et al. The effectiveness of manual physical therapy and exercise from mechanical neck pain: a randomized clinical trial. Spine. 2008;33:2371-2378; Table 1; with permission. **Study Design**: A case series of consecutive patients with cervical radiculopathy. **Background**: A multitude of physical therapy interventions have been proposed to be effective in the management of cervical radiculopathy. However, outcome studies using consistent treatment approaches on a well-defined sample of patients are lacking. The purpose of this case series is to describe the outcomes of a consecutive series of patients presenting to physical therapy with cervical radiculopathy and managed with the use of manual physical therapy, cervical traction, and strengthening exercises. **Case Description**: Eleven consecutive patients (mean age, 51.7 years; SD, 8.2) who presented with cervical radiculopathy on the initial examination were treated with a standardized approach, including manual physical therapy, cervical traction, and strengthening exercises of the deep neck flexors and scapulothoracic muscles. At the initial evaluation all patients completed self-report measures of pain and function, including a numeric pain rating scale (NPRS), the Neck Disability Index (NDI), and the Patient-Specific Functional Scale (PSFS). All patients again completed the outcome measures, in addition to the global rating of change (GROC), at the time of discharge from therapy and at a 6-month follow-up session. **Outcomes**: Ten of the 11 patients (91%) demonstrated a clinically meaningful improvement in pain and function following a mean of 7.1 (SD, 1.5) physical therapy visits and at the 6-month follow-up. **Discussion**: Ninety-one percent (10 of 11) of patients with cervical radiculopathy in this case series improved, as defined by the patients classifying their level of improvement as at least "quite a bit better" on the GROC. However, because a cause-and-effect relationship cannot be inferred from a case series, follow-up randomized clinical trials should be performed to further investigate the effectiveness of manual physical therapy, cervical traction, and strengthening exercises in a homogeneous group of patients with cervical radiculopathy. *J Orthop Sports Phys Ther* 2005;35:802-811. **Key Words**: *cervical spine, manipulation, mobilization, thoracic spine.* **FIGURE 3.3** Abstract from Cleland JA, Whitman JM, Fritz JM, Palmer JA. Manual physical therapy, cervical traction, and strengthening exercises in patients with cervical radiculopathy: a case series. *J Orthop Sports Phys Ther.* 2005;35:802-811; with permission. 8 ***QUESTION 5: Are the outcome measures relevant to the*** ***clinical question, and were they conducted in a clinically*** ***realistic manner?*** Study outcomes include a pain measure and a pain index, which may relate to June's goal to resume competitive swimming. Summary of Applicability Deciding the applicability of a study to your specific patient is a process of weighing the similarities and differences between the study participants and your patient and the intervention proposed and the feasibility of this intervention for you and your clinic. As you read research studies, you can keep track of answers to the questions regarding applicability by using the checklist in Table 3.3. **Part B: Determining** **Quality of an** **Intervention Study** Is This a Useful Study? Appraising the quality of a study focuses on understanding if the study was conducted with sufficient rigor that you can use the study results for making clinical decisions. Appraising study rigor is also referred to as appraising the *internal quality,* or *validity,* of a study.3 A study might have high applicability, but it may not have been conducted with sufficient rigor to be useful for clinical decisions. Appraisal requires the balancing of the applicability of a study and the quality of the study. Your goal is to have both applicability and quality rate as highly as possible (Fig. 3.4). The explanatory sections that follow are organized by questions to consider when appraising the quality of an intervention study. Following are questions 6 to 11, which are also included in Table 3.3. ***QUESTION 6:Were participants randomly assigned to*** ***intervention groups?*** Study Design **Randomized Controlled Trial or** **Randomized Clinical Trial** The randomized controlled trial or randomized clinical trial (RCT) is one type of research design. More designs are considered in other chapters; for example, designs that are most appropriate for questions of diagnosis (Chapter 5) and prognosis (Chapter 6). The RCT is often characterized as the gold standard of research designs for intervention studies. It is an experimental design in which treatments are compared. The RCT is considered one of the most valid designs to determine if a particular physical therapy treatment has a positive effect. Types of clinical questions that could be answered with an RCT include the following: 1\. Is manual therapy more effective than trunk stabilization for pain reduction and return to activity for people with acute low back pain? 2\. What is the efficacy of body weight--supported gait training as an intervention for school-age children with spastic diplegia? 3\. What is the efficacy of constraint-induced therapy as an intervention for individuals with a hemiplegia associated with stroke? Ideally, a random sample of subjects from the entire population of interest would be selected for study. For example, if the entire population of people who have sustained a stroke could be identified, then a sample from the entire population would be chosen randomly for study. Obviously it is not possible to locate or sample from an entire population, but a smaller sample might be selected randomly from a very large sample of people with a particular condition. Often in RCT designs, all *available* subjects are included in a study and then randomly assigned to treatment groups. If the sample is large enough (and this is key), then the random assignment ensures that all groups will be similar in the characteristics that are thought to affect outcome before the treatment begins. This similarity between groups is important to *reduce* *one potential source of bias* that might affect the outcome of the treatment. **Bias** in a study is any factor that makes you less certain that the results of the study are due to the intervention. Reducing bias increases your certainty that the results are due to the treatment. ***For example:*** If treatment group A has subjects with more severe disability than treatment group B, then group A might respond with greater change to treatment, regardless of type of treatment. **36 SECTION I Finding and Appraising Evidence to Improve Patient Care** **A** Applicability Search **Appraise:** **Intervention Studies** Integrate Evaluate **Step** **2 Step** **1** **3 Step** **4 Step** **5 Step** **Quality** Results Clinical Bottom Line **B** **C** **D** Clinical Question Low High 1\) **Applicability** to your question: Low High 2\) **Quality** of the study: One study can have very different rankings on the two scales. **Two Independent Scales for** **Research Evidence Appraisal:** Study 1 Study 2 Study 3 **FIGURE 3.4** Best evidence is a balance between applicability and quality. Conversely, treatment group A might respond more poorly to an intense treatment because the subjects may not have sufficient endurance for the treatment. Randomly assigning subjects to groups should even out the groups in terms of severity of disability and reduce this source of bias. RCTs are typically designed to evaluate the efficacy of an intervention.4,5 Efficacy refers to the effect of treatment under highly controlled conditions. This is in contrast to a study of effectiveness, in which the treatment would be evaluated under more typical clinical conditions. Effectiveness speaks to what is likely to happen given a treatment versus efficacy, which determines what could happen in the best possible environment. These are each important but very different questions. Stratification Within RCTs Randomization is most effective in reducing potential bias if the sample size is large; however, recruiting enough subjects is often challenging. Stratification of subjects is used in the RCT in situations in which there are potential characteristics that are thought or known to affect outcomes and it is not certain that a sufficient number of subjects with this characteristic will be recruited. For example, if previous research suggests that men and women respond differently to a particular treatment, but the problem is less common in women than it is in men, it may be difficult to recruit enough women to be sure the groups will be similar according to gender. The groups could be stratified on gender and then males and females randomized separately into treatment groups. This gives you more precise applicability of the research to your male and female patients. Figure 3.5 includes an abstract of a study in which subjects are stratified based on locomotor impairment defined as speed of walking. QUESTION 7: Is the sampling procedure (recruitment strategy) likely to minimize bias? Samples for Studies The value of randomization to reduce one source of bias and the need to have subject groups as similar as possible at the beginning of a study have been discussed. There are additional sampling issues that should be considered to minimize bias in the sample of participants chosen for a study. Sample Size Study samples in physical therapy are often small. Sometimes samples are too small for the study results to be truly believable (valid). It is difficult and costly to obtain large numbers of subjects with certain types of injury, movement issues, or medical CHAPTER 3 Critically Appraise the Applicability and Quality of an Intervention Research Study 37 FIGURE 3.5 Abstract from Duncan PW, Sullivan KJ, Behrman AL, et al. Protocol for the Locomotor Experience Applied Post-stroke (LEAPS) trial: a randomized controlled trial. BMC Neurol. 2007;7:39; with permission. Protocol for the Locomotor Experience Applied Post-stroke (LEAPS) trial: a randomized controlled trial8 Pamela W Duncan, Katherine J Sullivan, Andrea L Behrman, Stanley P Azen, Samuel S Wu, Stephen E Nadeau, Bruce H Dobkin, Dorian K Rose, Julie K Tilson for the LEAPS Investigative Team Abstract Background: Locomotor training using body weight support and a treadmill as a therapeutic modality for rehabilitation of walking post-stroke is being rapidly adopted into clinical practice. There is an urgent need for a well-designed trial to determine the effectiveness of this intervention. The objective of the Locomotor Experience Applied Post-Stroke (LEAPS) trial is to determine the effectiveness if there is a difference in the proportion of participants who recover walking ability at one year post-stroke when randomized to a specialized locomotor training program (LTP), conducted at 2- or 6-months post-stroke, or those randomized to a home based non-specific, low intensity exercise intervention (HEP) provided 2 months post-stroke. We will determine if the timing of LTP delivery affects gait speed at 1 year and whether initial impairment severity interacts with the timing of LTP. The effect of number of treatment sessions will be determined by changes in gait speed taken pretreatment and post-12, -24, and -36 sessions. Methods/Design: We will recruit 400 adults with moderate or severe walking limitations within 30 days of stroke onset. At two months post-stroke, participants are stratified by locomotor impairment severity as determined by overgound walking speed and randomly assigned to one of three groups: (a) LTP-Early; \(b) LTP-Late or (c) Home Exercise Program-Early. The LTP program includes body weight support on a treadmill and overground training. The LTP and HEP interventions are delivered for 36 sessions over 12 weeks. Primary outcome measure includes successful walking recovery defined as the achievement of a 0.4 m/s gait speed or greater by persons with initial severe gait impairment or the achievement of a 0.8 m/s gait speed or greater by persons with initial moderate gait impairment. LEAPS is powered to detect a 20% difference in the proportion of participants achieving successful locomotor recovery between the LTP groups and the HEP group, and a 0.1 m/s mean difference in gait speed change between the two LTP groups. Discussion: The goal of this single-blinded, phase III randomized clinical trial is to provide evidence to guide post-stroke walking recovery programs. Each of the three groups in this study will have an equal number of subjects with moderate walking limitations and with severe walking limitations. Patients may respond differently to locomotor training depending on the severity of their walking limitations. problems. For example, patient characteristics for a hypothetical study are listed in Table 3.2. If your patient is a black female with an affected right side and age 62, this study might be helpful to your clinical decisions. Similarly, if your patient is an Hispanic male with an affected left side and age 83, this study might also be useful for clinical care. The authors of this hypothetical study believed that gender, race, age, and side affected would have an impact on the results of the study. By including all of these characteristics, the applicability of the study is increased. However, sufficient numbers of subjects are needed in each category in Table 3.2 to make the study valid. Imagine that the study in Table 3.2 has 8 white males with an affected left side who are in the 60-to-70 age range and 232 black males with an affected right side also in this age range. This imbalance in the number of subjects in the study creates problems in interpreting the results of the study and applying the results to specific patients. **Study Power** Without sufficient numbers of subjects, a study may lack power. Power is a statistical term.6,7 The power of a statistical test is the likelihood that the test will detect a difference between groups if there is a difference. The bigger the difference between treatment groups, the more likely that the difference will be statistically significant. The size of a sample affects power. If a sample is too small, it is possible for the results of a study to suggest that there is no difference between treatments when in fact there is a difference. The study conclusions are not helpful for clinical decisions if the study lacks power. Statistical quality is discussed more thoroughly in Chapter 4, but the important message here is that if samples are too small (study is underpowered) it may not be a good enough study to help you make decisions. **Recruiting a Study Sample** There are different methods used to recruit participants for a study. For example, a study of physical therapy for postsurgical anterior cruciate ligament (ACL) repair might include all patients who are referred to one physical therapy clinic. A consecutive sample6 (Fig. 3.6) includes all patients referred to this clinic with this problem. Because people with many different characteristics (e.g., age, athletic ability, gender) are referred for physical therapy post ACL surgery, the sample from this clinic might represent this broad sampling. When consecutive sampling is used, *every* patient post ACL surgery who is referred to the clinic is a potential subject. Each patient is then considered in terms of inclusion or exclusion criteria for the study. Selective sampling (see Fig. 3.6) is an alternative method. If orthopedic surgeons and other professional colleagues are asked to refer their patients post-ACL surgery, then most likely a selective sample is recruited. Not all patients will be referred. The surgeons or colleagues may suggest the study only to patients who they believe are likely to participate, or they might remember to refer patients after first hearing about the study, but forget referrals later on. Selective sampling is less ideal than consecutive sampling, but it is common and practical. **38 SECTION I Finding and Appraising Evidence to Improve Patient Care** TABLE 3.2 **Example Patient Characteristics** **PATIENT CHARACTERISTICS ACTIVITIES OF DAILY LIVING WALKING SPEED QUALITY OF LIFE** Gender Male Female Race White Black Hispanic Other Side Affected Left Right Age 60--70 71--80 81--90 **Testers and Treaters: The Importance** **of Masking** ***QUESTION 10:Was blinding/masking optimized in the*** ***study design? (evaluators, participants, therapists)*** The person giving the treatment during an intervention study should not be the person who measures the patients' results from the intervention. The "treaters" should not be the "testers." Ideally, the testers should not know which treatment a patient received during the study and they should know as little as possible about the study goals (Fig. 3.8). This is referred to as masking or blinding.6 Neither the treaters nor the testers may consciously bias the results, but their opinions and clinical experiences with specific treatments may influence their measurements. If the treating therapist also measures the results of the intervention, this is a serious threat to the quality of the results. It reduces your certainty that the study results were due to the specific intervention and thus reduces the usefulness of the results for your clinical decisions. **Equivalent Treatment** ***QUESTION 11: Aside from the allocated treatment, were*** ***groups treated equally?*** When two or more treatments are compared or when a treatment is compared with that of a control group, all other interventions must be stopped or held constant for the duration of the study. For example, if some subjects in a treatment group also participated in another type of intervention (e.g. adding therapy, taking medications, or seeking alternative treatments), this reduces the certainty that the results of the study were due to the original intervention. Patients may be taking medications that should not be stopped during a study, but these must be reported and held as constant as possible during the study duration. Study participants are asked to refrain from other interventions during the study, but this should also be monitored periodically during a study, especially if study duration is lengthy. Treatments should also be equivalent in intensity (Fig. 3-9). One type of treatment may involve longer periods of therapy; for example, manual therapy plus exercise versus exercise alone. The length of therapy and the intensity or dose (once a week, twice a week, etc.) must be equivalent. An equal amount of exercise may be used for both study groups and while one group receives manual therapy, the exercise-only group may receive an educational session or other form of intervention to make the time in therapy equivalent. **Summary of Research Study Quality** The above content is summarized as a checklist in Table 3.3. These questions are used to appraise the *quality* of a study. As you read the study, you can keep track of answers to the questions by checking "Yes" or "No." This is a partial checklist, **40 SECTION I Finding and Appraising Evidence to Improve Patient Care** I hope the research shows that my treatment is best! **FIGURE 3.8** The therapist giving treatment in a research study should not measure outcomes in the study. Exercise \_ Manual Therapy \_ 90 minutes Exercise \_ Education \_ 90 minutes **Treatment A** **Treatment B** **FIGURE 3.9** Treatments A and B must be equivalent in length of treatment. **CHAPTER 3 Critically Appraise the Applicability and Quality of an Intervention Research Study 41** TABLE 3.3 **Key Questions to Determine an Intervention Research Study's *Applicability*** **and *Quality*** **WHERE TO FIND** **QUESTION YES/NO THE INFORMATION COMMENTS AND WHAT TO LOOK FOR** 1\. Is the study's purpose \_\_ Yes Introduction (usually The study should clearly describe its purpose and/or relevant to my clinical question? \_\_ No at the end) hypothesis. Ideally, the stated purpose will contribute to answering your clinical question. 2\. Is the study population \_\_ Yes Results section The study should provide descriptive statistics about (sample) sufficiently similar \_\_ No pertinent study population demographics. Ideally, to my patient to justify the the study population would be relatively similar to expectation that my patient your patient with regard to age, gender, problem would respond similarly to severity, problem duration, co-morbidities, and the population? other sociodemographic and medical conditions likely to affect the results of the study. 3\. Are the inclusion and exclusion \_\_ Yes Methods section The study should provide a list of inclusion and criteria clearly defined and would \_\_ No exclusion criteria. Ideally, your patient would have my patient qualify for the study? characteristics that meet the eligibility criteria or at least be similar enough to the subjects. Remember, you will not find "the perfect study"! 4\. Are the intervention and \_\_ Yes Methods (some The study should clearly describe the treatment comparison/control groups \_\_ No post-study analysis regimen provided to all groups. Ideally, the receiving a realistic intervention? about the intervention intervention can be reproduced in your clinical may be found in the setting and the comparison/control is a realistic Results) contrasting option or well-designed placebo. Consider the quality of the dose, duration, delivery method, setting, and qualifications of the therapists delivering the intervention. Could you implement this treatment in your setting? 5\. Are the outcome measures \_\_\_ Yes Methods The study should describe the outcome measures relevant to the clinical question \_\_\_ No used and the methods used to ensure their and were they conducted in a reliability and quality. Ideally, the outcome measures clinically realistic manner? should relate to the clinical question and should include measures of quality of life, activity, and body structure and function. For diagnostic studies, it is important that the investigated measure be realistic for clinical use. 6\. Were participants randomly \_\_ Yes Methods The study should describe how participants were assigned to intervention groups? \_\_ No assigned to groups. **Randomization** is a robust method for reducing **bias.** Computerized randomization in which the order of group assignment is concealed from investigators is the strongest method. 7\. Is the sampling procedure \_\_ Yes Methods The study should describe how and from where (recruitment strategy) likely \_\_ No participants were recruited. **Consecutive** to minimize bias? **recruitment,** in which any participant who meets eligibility criteria is invited to join the study, is the strongest design. Studies in which the authors handpick participants may demonstrate the best effects of a specific treatment, but they will lack applicability across patient types. 8\. Are all participants who \_\_ Yes The study should describe how many participants entered the study accounted for? \_\_ No were initially allocated to each group and how many finished the study. Ideally, \>80% of participants complete the study and the reason for any participants not finishing is provided. Continued **42 SECTION I Finding and Appraising Evidence to Improve Patient Care** TABLE 3.3 **Key Questions to Determine an Intervention Research Study's *Applicability*** **and *Quality*---cont'd** **WHERE TO FIND** **QUESTION YES/NO THE INFORMATION COMMENTS AND WHAT TO LOOK FOR** 9\. Was a comparison made \_\_ Yes Methods will describe The study should make comparisons between between groups with preservation \_\_ No the planned analysis. groups (not just a comparison of each group to of original group assignments? Results will report itself). Ideally, an **intention-to-treat analysis** is findings. conducted to compare groups' participants according to their original group assignment. 10\. Was blinding/masking \_\_ Yes Methods The study should describe how/if **blinding** was optimized in the study design? \_\_ No used to reduce bias by concealing group assignment. (evaluators, participants, Most physical therapy--related studies cannot blind therapists) therapists or participants due to the physical nature of interventions. Bias can be reduced, however, by blinding the evaluators (who conduct the outcome measures). 11\. Aside from the allocated \_\_ Yes Methods will describe The study should describe how equivalency, other treatment, were groups treated \_\_ No the treatment plan. than the intended intervention, is maintained equally? Results may report between groups. Ideally, all participants will have adherence to plan. exactly the same experience in the study except for the difference between interventions. with the remaining questions to appraise study quality included in a checklist in Chapter 4. n**Research Notation** When you first read a study, you want to understand *what* was done and then determine *how* it was done. Research notation, first described by Campbell and Stanley,7 is a helpful shorthand to diagram the design of an intervention study and give you the big picture of the overall study. Research notation will be used to diagram what was done in the example studies and identify the type of design used in the study. Different clinical questions are best answered with specific study designs. For example, the RCT is an excellent design for intervention research but not for prognostic studies. Thus research notation is used to depict intervention studies. There are four symbols used in research notation with the following definitions: **R** subjects were randomly assigned to treatment groups **\_ \_ \_** subjects were not randomly assigned **X** intervention **O** observation (time of measurement) **Using Research Notation for Example** **Study Designs** The studies diagrammed in Figure 3.10, Examples 1 and 2, have less potential bias than the study diagrammed in Example 3. In Example 1 two treatments are compared, and in Example 2 one treatment is compared with a no-treatment condition. The design in Example 1 helps to determine which treatment might have a greater effect. The design in Example 2 helps to determine if treating the patient with a particular treatment is better than doing nothing. If there are positive results from Example 3, you may determine that subjects improved, but it does not help you to understand how to choose among treatments or if time alone might have been sufficient for the subjects to improve. Studies in which a control (no treatment) or comparison (an alternate treatment) are used have higher quality than studies with only one treatment group. n **SELF-TEST 3.1** Using research notation, diagram the study in the abstract found in Figure 3.2. Compare your answers with those in Answers to Self-Test. **CHAPTER 3 Critically Appraise the Applicability and Quality of an Intervention Research Study 43** 2\) Each group was observed once before treatment at ***baseline*** (subject measurements were taken before treament) 3\) Two treatments were given. Each group received one type of treatment 1\) Subjects were randomly assigned to one of the two intervention groups 4\) Each group was observed once after treatment at ***follow-up*** (subject measurements were taken after treatment) **R O X** **O X** **O** **O** Treatment may have been given once or multiple times. The X merely indicates that treatment was given but not the type or amount of treatment. Research notation does not specify what measures were taken at each observation point. We have a general diagram of what was done in this study, but no specifics about treatments or outcome measures. 2\) Each group was observed once before treatment (subject measurements were taken before treatment) 3\) One group received treatment and the other group received no treatment 1\) Subjects were randomly assigned to one of the two groups 4\) Each group was observed twice after treatment (subject measurements were taken after treatment) **R O X** **O** **O O** **O O** 2\) Subjects were observed once before treatment 3\) Only one treatment was given; the absence of a second row indicates that there was no control or group comparison 1\) Subjects were not randomized and there is only one group 4\) Subjects were observed once after treatment **\-\-\-- O X O** **Design Example 3: This notation indicates that:** **Design Example 2: This notation indicates that:** **Design Example 1: This notation indicates that:** **FIGURE 3.10** Diagrams of example research designs using research notation. S U M M A RY Appraising the quality of research evidence: Once you have determined that a research study is applicable to your patient, you will appraise the quality of the research. You will determine if a study has been conducted with sufficient rigor for you to use the results in the care of your patient. Appraising the rigor or *internal quality* of a research study includes an understanding of sampling of subjects, measurement, and sources of bias in a study. Appraising the applicability of research evidence: After identifying your clinical question, searching for evidence, and then locating research evidence, you determine if the research is appropriate for your particular patient. The *applicability* of research is a first step in appraising research. Applicability is determined by comparing the study participants' characteristics to your patient (e.g., age, rehabilitation potential, gender). Then you determine if the intervention is appropriate for your patient and if you have the skills and necessary equipment in your clinic to implement the treatment intervention **1.** State the three most important characteristics of a highquality intervention study. **2.** Why should the person who measures the outcomes in an intervention study not administer the intervention? **3.** What is more relevant to the application of a study to your clinical question, the applicability of the study or the study quality? Defend your answer.