Chapter 10: Planning and Evaluating Applied Behavior Analysis Research PDF

Chapter 10: Planning and Evaluating Applied Behavior Analysis Research Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved Importance of Individual Subject Enables ap...

Chapter 10: Planning and Evaluating Applied Behavior Analysis Research Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved Importance of Individual Subject Enables applied behavior analysts to discover and refine effective interventions for socially significant behaviors Contrasted with groups-comparison approach Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved Groups-Comparison Experiment Randomly selected pool of subjects from relevant population Divided into experimental and control groups Pretest, application of independent variable to experimental group, and posttest Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Group Data Not Representative of Individual Performance Individuals within a group could stay the same or decrease, while the improvement of others could make it appear as overall average improvement To be most useful, treatment must be understood at an individual level Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Group Data Masks Variability Hides variability that occurs within and between subjects Statistical control should not be a substitute for experimental control To control effects of any variable, must either hold it constant or manipulate it as an independent variable Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved Absence of Intrasubject Replication Power of replicating effects with individuals is lost Many applied situations in which overall performance of group is socially significant When group results don’t represent individuals, should supplement the data with individual results Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Importance of Flexibility in Design An effective researcher must actively design each experiment so that it achieves its own unique design Good experimental design is any independent variable manipulation that produces data that convincingly addresses the research question The book presents analytic tactics in design form Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Experimental Designs Often designs entail a combination of analytic tactics Component analysis of elements Infinite number of possible designs with different combinations Most effective use ongoing evaluation of data from individuals to employ baseline logic of prediction, verification, and replication Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Internal Validity Experiments that demonstrate clear functional relations have high degree of internal validity Experimental control refers to all relevant variables Steady state responding as evidence Confounding variables are threats to internal validity Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Subject Confounds Maturation: changes in subject over course of experiment Repeated measurement controls and detects uncontrolled variables Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Setting Confounds Studies in natural settings are more prone to confounding variables than in controlled laboratories If change in setting occurs, should then hold new conditions constant until steady state responding is observed Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Measurement Confounds Observer drift or bias Keeping observers naïve to expected outcomes can reduce observer bias Must maintain baseline conditions long enough for reactive effects to run their course and then obtain stable responding Could use intermittent probes except when practice effects would be expected Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Independent Variable Confounds Placebo control separates effects produced by subject’s perceived expectations Double-blind control eliminates confounding by subject expectations, teacher and parent expectations, differential treatment by others, and observer bias Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Treatment Integrity Similar to procedural fidelity Extent to which the independent variable is implemented or carried out as planned Low treatment integrity makes it very difficult to confidently interpret experimental results Treatment drift: when application of independent variable in later phases differs from original application Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Precise Operational Definition A high level of treatment integrity requires a complete, precise operational definition of treatment procedures Define in 4 dimensions: verbal, physical, spatial, and temporal Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Simplify, Standardize, and Automate Simple, precise treatments are more likely to be consistently delivered Simple, easy-to-implement techniques are more likely to be used and socially validated Experimenters should standardize as many aspects as possible and practical If possible without compromise, could use an automated device to deliver independent variable Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Training and Practice Train or provide practice for individual who will conduct the experimental sessions Could provide a detailed script, verbal instructions, modeling, or performance feedback Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Assessing Treatment Integrity Collect treatment integrity data to measure how the actual implementation of the conditions matches the written methods Observation and calibration give the researcher the ongoing ability to use retraining and practice to ensure high treatment integrity Reduce, eliminate, or identify the influence of as many potentially confounding variables as possible Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Social Validity Includes the social significance of the target behavior, the appropriateness of the procedures, and the social importance of the results Usually assessed by asking direct and indirect consumers Consumer satisfaction Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Social Importance of Behavior Change Goals To determine socially valid goals: – Assess the performance of persons considered competent – Experimentally manipulate different levels of performance to determine which produces optimal results Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Social Importance of Interventions Rating scales and questionnaires for obtaining consumers’ opinions on acceptability of interventions Examples: – Intervention Rating Profile – Treatment Acceptability Rating Form Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Social Importance of Behavior Changes Methods for assessing outcomes: – Compare subject’s performance to a normative sample – Use standardized assessment instrument – Ask consumers to rate social validity of performance – Ask experts to evaluate subject’s performance – Test subject’s new performance in natural environment Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Normative Sample Not limited to posttreatment comparisons Compare subject’s behavior to ongoing probes of behavior of normative sample to provide ongoing measure of improvement and how much is still needed Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Consumers and Experts Most frequently used method for assessing social validity is to ask consumers Experts can be called upon to judge the social validity of some behavior changes Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved Standardized and Real-World Tests Example of standardized test: Self-Injury Trauma Scale (SITS) Real-world test in the natural environment provides direct assessment of social validity Also exposes subject to naturally occurring reinforcement, which may promote maintenance and generalization Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  External Validity Degree to which a functional relation in an experiment will hold under different conditions A matter of degree, not all-or-nothing Those with greater degrees of generality, make greater contribution to applied behavior analysis Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  External Validity and Groups- Design Research There is nothing in the results of a groups- design experiment that can have external validity Unable to provide data that lead to improved practice in education Groups-design is effective in large-scale evaluations Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  External Validity and Applied Behavior Analysis Generality of findings in ABA is assessed, established, and specified through replication of experiments Two major types of scientific replication: direct and systematic Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Direct Replication Duplicates exactly the conditions of an earlier experiment Intrasubject direct replication: uses same subject to establish reliability of functional relation Intersubject direct replication: uses different but similar subjects to determine generality Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Systematic Replication Researcher purposefully varies one or more aspects of earlier experiment Can demonstrate reliability and external validity of earlier findings Can alter any aspect: subjects, setting, administration of independent variable, or target behaviors Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Evaluating Applied Behavior Analysis Research Questions to ask in evaluating the quality of research in applied behavior analysis fall under 4 categories: – Internal validity – Social validity – External validity – Scientific and theoretical significance Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Internal Validity Must decide whether functional relation has been demonstrated Requires close examination of measurement system, experimental design, and the researcher’s control of potential confounds Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Evaluating Internal Validity Definition and measurement of dependent variable Graphic display Meaningfulness of baseline conditions Experimental design Visual analysis and interpretation Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Visual Analysis and Interpretation Factors that favor visual analysis over tests of statistical significance in ABA: – Want to see socially significant behavior change, not statistically significant – Good for identifying variables that produce strong, large, and reliable effects – Accepting statistical analysis as evidence of functional relation may cause researcher not to experiment further – Tests of statistical significance may cause data sets to conform, losing flexibility in design Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Errors Type I error: when researcher concludes that independent variable had effect on dependent variable, when it did not Type II error: when researcher concludes that independent variable did not have effect on dependent variable, when it did Visual analysis leads to less Type I and more Type II errors Statistical analysis leads to more Type I and less Type II errors Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Social Validity Independent variable should be assessed in terms of its effects on dependent variable, as well as social acceptability, complexity, practicality, and cost Consider maintenance and generalization of behavior change in evaluation of a study Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  External Validity To effectively judge external validity, compare a study’s results with those of other relevant published research Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Theoretical Significance and Conceptual Sense Evaluate a study in terms of its scientific merit Look at its contribution to the advancement of the field “knowledgeable reproducibility” Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved  Need for More Thorough Analyses Need for more conceptual understanding of the principles that underlie successful demonstrations of behavior change Readers should consider the technological description, the interpretation of results, and the level of conceptual integrity in experimental reports Cooper, Heron, and Heward Copyright © 2007 by Pearson Education, Inc. Applied Behavior Analysis, Second Edition All rights reserved