Lecture 11 & 12 PSYCH2018 Lecture Notes PDF

# Chapter 10. Non-Experimental Design II: Observational and Archival Methods ## Chapter Objectives * Distinguish between naturalistic and participant observation methods * Articulate the problems that can occur in observational research and how researchers address those problems * Explain how them...

# Chapter 10. Non-Experimental Design II: Observational and Archival Methods ## Chapter Objectives * Distinguish between naturalistic and participant observation methods * Articulate the problems that can occur in observational research and how researchers address those problems * Explain how thematic analysis can be a tool for evaluating qualitative data * Define archival research and explain why archival research is non-experimental * Describe the advantages and limitations to archival research * Describe methods of analyzing data from observational and archival research * Explain how meta-analysis can be used to evaluate replication of research results ## Prof. Ryan Fun Fact Observational research in anthropology: * **emic:** insider perspective * Attempts to understand a phenomenon from the perspective of those directly involved * Similar to participant observation * **etic:** outsider perspective * Uses existing theories to describe/explain observed phenomenon * Similar to naturalistic observation ## Naturalistic Observation * Describing behaviors in natural settings * Observer is unobtrusive, or * Habituation assumed ### Examples: * Snack selection at movie theaters * Gender differences in fighting behaviors at a bar * Helping behaviors in a preschool setting ## Participant Observation * Experimenter joins group being observed * e.g., Festinger's study of a cult (Box 10.1) * Data recording problems * Ethical issues and reactivity * Experimenters changing the group ## Challenges Facing Observational Methods * Absence of control * Falsification of strong claims possible * Observer bias * Use of behavior checklists * Interobserver reliability * Time and event sampling * Participant reactivity * Use of unobtrusive measures helps * Ethics * Consent and privacy issues (Box 10.2) ## Young Ryan Moment What happens when faith and science mix? * Interest in glossolalia * Recorded (and transcribed) speech in public area * Follow-up questions after the event to gather data * Identified key phonemes in “tongues” & compared to phonemes in primary language * Shared findings with community leaders and my psycholinguistics professor Where did I make a mistake? ## Research Example 29 * Naturalistic observation in a science museum * Consent obtained (unusual in observational research) * Event sampling used * Results: Parents (Dads and Moms) explain science concepts more to their sons than to their daughters (Fig. 10.1) ## Research Example 30 "Covert" participant observation at a homeless shelter * Discovering “identity maintenance strategies" in homeless individuals * Experimenter served as a volunteer at homeless shelter * She had already done so for 2 years prior * Experimenter kept a journal of her observations for 3 months * Results: How individuals interacted with others and how they thought of themselves depended upon how long they were homeless ## Analyzing Qualitative Data ### Thematic analysis * Method of identifying patterns of responses in qualitative data * Six steps: 1. Get familiar with the data 2. Code 3. Search for themes 4. Review themes 5. Define and name themes 6. Write report ## Archival Research * Archival data: * Data previously collected for some other purpose * Often undergoes content analysis * Susceptible to missing data and bias, but no reactivity ### Research Example 31 * IV: Patient recovering room * Experimental: Pleasant view of park * Nonequivalent control: Brick wall * DV: Recovery & other factors (better for room with a view) ## Analyzing Archival Data * Factor analysis: * Identifies what tests/items cluster to form factors * Meta-analysis: * A special case of archival research * Analysis of effect sizes across multiple studies all related to a similar topic * Two main questions can be answered: * Is the effect consistent across studies? * If the effect is consistent, what is the size of the effect? ## Example: Factor Analysis The chart contains a table with data on factor loadings and communalities. ## Example: Meta-Analysis The chart represents a meta-analysis of the treatment effect of interventions for promoting smoke alarm ownership and function. The analysis shows that there is a significant treatment effect (OR = 1.21; 95% CI = 0.89, 1.64). ## Summary * Observational research methods are fairly benign, unobtrusive methods for obtaining information about various psychological phenomena. * Observational methods are used to assess natural behaviors either in their natural setting or in a lab setting. * Thematic analysis can be used to evaluate qualitative data from non-experimental designs. * Archival research is used to analyze previously collected data to answer an empirical question. * Factor analysis and meta-analysis can be used with archival research. # Chapter 11. Quasi-Experimental Designs and Applied Research ## Key Terms & Concepts * Quasi-experimental design * Nonequivalent control group design * Regression to the mean * Interrupted time series * Program evaluation * Needs analysis * Formative & summative evaluation * Cost-effectiveness analysis ## Beyond the Laboratory * Dual functions of applied research: 1. Solves real-world problems 2. Increases basic knowledge; evaluates theory ### Research Example 33 * IV: Color code of nutrition labels * Illustrating function #1 - improved non-dieters' use of color-coded labels, both in terms of evaluation of health quality and food consumption. * Illustrating function #2 - supported importance of decision-making processes in health-related behaviors. ## Applied Psychology in Historical Context Early experimentalists, trained in basic research, felt pressures to provide “relevance.” * **Harry `[illegible]`** worth, hired to examine the effects of `[illegible]` a-Cola (Box 11.1) on `[illegible]` for sleep & addictive? **Prof. Ryan warning**: **VALIDITY ALERT!!** * Methodological: * Counterbalancing * Double blind and placebo control * Results: * No adverse effects of caffeine, except on sleep ## Design Problems in Applied Research * Ethical dilemmas: * Consent, privacy, potential coercion * Trade-off between internal and external validity * Internal validity may suffer * Problems unique to between-subjects designs: * Can be difficult to create equivalent groups * Problems unique to within-subjects designs: * Uncontrolled order effects, attrition ## Quasi-Experimental Designs * No causal conclusions * Less than complete control * No random assignment ### From prior chapters * Single-factor nonequivalent groups designs * *Ex post facto* factorial designs * P x E factorial designs * All the correlational research ## Nonequivalent Control Group Designs * Typically (but not necessarily) include pretests and posttests. **Experimental group** 01 T 02 **Noneq. control group** 01 02 * Random assignment to groups is not possible for practical reasons. * Two groups may initially be different at O1. ## Quasi-Experimental Designs ### Nonequivalent control group designs * Fig. 11.2 * Best outcome: d (lower right) ## Nonequivalent Control Group Designs ### Regression to the mean and matching * Matching on pretest may produce: * Experimental group that scores higher than population on pretest * Control group that scores lower than population on pretest * Both groups may regress to mean on posttest, masking any real change due to treatment. ## Nonequivalent Control Group Designs With Pretests ### Research Example 34 * IV1: Whether or not Play Streets is implemented in city streets * Nonequivalent groups: Streets are in different city neighborhoods * IV2: Before/after intervention * DVs: Amount of physical activity. * Fig. 11.4 ## Nonequivalent Control Group Designs Without Pretests ### Research Example 35 * IV: Living distance from SF earthquake * Experimental group: California * Nonequivalent control group: Arizona * No pretests (difficult to predict earthquakes) * DV: Nightmare frequency. * Results * California > Arizona. * Ruled out alternative explanation that those in California would always have more earthquake nightmares ## Interrupted Time Series Designs * Useful for evaluating overall trends * Basic design: O1 O2 O3 O4 O5 T 06 07 08 O9 010 * Fig. 11.6 * Outcomes: * Best outcome: d (lower right) ## Interrupted Time Series Designs ### Research Example 36 * Effect of incentive plan on productivity * Ruled out effects of history, instrumentation, and subject selection * Fig. 11.7 ## Interrupted Time Series Variations * Variations on the basic time series design: * Add a control group * 01 02 03 04 05 T 06 07 08 O9 010 * 01 02 03 04 05 06 07 08 09 010 * Add a "switching" replication * **Second treatment, but at a different time** * 01 02 03 T 04 05 06 07 08 09 10 * 01 02 03 04 05 06 07 T 08 09 010 * Add a second DV, not expected to be influenced by the program. * Fig 11.8. Three strike law? Adding misdemeanor data to felony data ## Program Evaluation * Reforms as experiments (Box 11.2) * Connecticut speeding study: Regression plus some degree of effectiveness with control states comparison (Fig. 11.9). * Campbell? encouraged “experimental” administrators, rather than “trapped” administrators. ## Needs Analysis * Planning for programs: * Census data * Surveys of available resources * Surveys of potential users * Key informants, focus groups, community forms ### Research Example 37 * Healthy behaviors in the workplace (DuPont) * Examined employee health data * Surveyed existing company programs * Surveyed employee knowledge of healthy behavior ## Program Evaluation * Monitoring programs - Formative evaluation: * Evaluating program while in progress. * Implemented as planned? * Program audit (how something is used) * Pilot study * Evaluating outcomes - Summative evaluation: * Program effectiveness * More threatening than formative evaluation * Use of quasi-experimental designs * Failure to reject null can be a useful outcome * New programs have to prove themselves. ## Cost-effectiveness Analysis * Weighing costs: * Program cost * Two equally effective programs, but may differ in costs * Most cost-effective wins * Potential cost also relevant for initial decision about need * Research Example 37 (continued): Comparing worksite wellness programs * Large-scale fitness centers not worth the cost. * Strategy emphasizing social support and follow-up was cost effective ## Prof. Ryan’s I/O Days In an electronics manufacturing company * IV: Training Program (0 hours, 4 hours, 8 hours, 24 hours) * DV: Average Error Rate in First Month * Results: 0 (20) < 4 (12) < 8 (9) = 24 (8) So, with 8 and 24 being equal….. But what about cost? * Training = $20/hour (4 = $80, 8 = $160, difference of $80) * Are 3 errors (12 errors - 9 errors) worth more or less than $80? * Each Error = $573 ## Summary * Applied research sheds light on causes and solutions to real-world problems. * Quasi-experimental designs are used when participants cannot be randomly assigned to groups and include: nonequivalent control group and interrupted time series designs. * Program evaluation is used to assess the effectiveness of large-scale programs (e.g., Head Start).