Psychology 306 Chapter 8: Between-Subjects Design PDF

PSYCH 306 Chapter 8: Between-Subjects Design Page 1  Outline The between-subjects design (also called "between-groups design“ or “independent groups design”) Systematic and non-systematic variance – The F-ratio Confounding variables...

PSYCH 306 Chapter 8: Between-Subjects Design Page 1  Outline The between-subjects design (also called "between-groups design“ or “independent groups design”) Systematic and non-systematic variance – The F-ratio Confounding variables – Individual differences & environmental variables Randomization Additional threats to internal validity Advantages/disadvantages Page 2 Between-Subject Designs A different group of participants is assigned to each condition Each group receives a different experimental treatment (value of the IV) and the groups are compared Group 1 (Violent Cartoons) Group 2 (Cartoons with no violence) Page 3 Between-Subject Design Key element: separate groups of participants are used for the different conditions Their data (on the DV) is compared across groups to look for differences Page 4 Between-Subjects Design Because participants experience only one level of the IV, there is 1 score on the DV per participant These are independent measures (also called independent-measures experimental design) Some independent variables can ONLY be measured in a between-subjects design Examples: IV = age; gender Other independent variables can be measured in a between-subject or within-subject design Examples: IV = teaching method; video condition Page 5  Between-Subjects Designs 15-month-old infants watch an adult persist for 30 s in opening objects in Effort or No-effort conditions Control condition: Baseline (no adult) Infants given container with button to press Between-groups Design (34 infants/condition) Infants who see the adult persist in the Effort condition show more attempts to open their container Leonard et al, 2017, Science Why a between-groups design? Infants have limited attention span, experiment often ends earlier than planned Between-subjects design allows 1 condition/infant (limited time commitment) Page 6 Between-Subjects Design To look for an effect of your IV, you compare the mean scores (DV) for each group 1. The difference in the DV between groups is referred to as SYSTEMATIC VARIANCE 2. Scores also vary within groups (individual differences occurring by chance) called NON-SYSTEMATIC VARIANCE Non-systematic variance is an important source of error and therefore must be minimized Can use these two sources of variability (variance) to calculate our test-statistic – Treatment index; F-ratio Page 7  Between-Subjects Design Group 1 Group 2 Music Lessons No Lessons SYSTEMATIC DV = VARIANCE NON-SYSTEMATIC NON-SYSTEMATIC 14 10 Verbal VARIANCE VARIANCE Ability 12 11 score 18 9 16 7 15 10 Page 8 Systematic Variance 1. Between-subjects (systematic) variance: differences among the means of different treatment groups No cartoon violence Moderate cartoon violence High cartoon violence # aggressive acts = 5 # aggressive acts = 11 # aggressive acts = 14 Compare differences between means These differences in means could be due to: – Treatment effects (being exposed to violent cartoons) – Effects of chance factors (experimental error) Page 9 Systematic Variance Hypothetical: If we treated the groups the exact same way, would you expect to see the exact same scores on aggression? No cartoon violence No cartoon violence No cartoon violence # aggressive acts = 5 # aggressive acts = 7 # aggressive acts = 3 No… impossible to perfectly match groups, therefore there is always some error, and always some differences between means Experimental error: all chance factors not controlled for – e.g., individual differences, variations in testing environment Page 10  Systematic Variance Between-subjects (systematic) variance has 2 components: (treatment effects) + (experimental error) No cartoon violence Moderate cartoon violence High cartoon violence # aggressive acts = 5 # aggressive acts = 11 # aggressive acts = 14 Must determine whether differences between the group means are due to experimental error alone, or experimental error + treatment effects Page 11  Non-Systematic Variance Within-group (non-systematic) variance: any differences (variation) between subjects who are treated alike Within any given treatment group, all subjects have been treated identically, and should therefore all have same value for DV Any variability can only be a result of chance factors Non-systematic (within-group) variance = experimental error No cartoon violence Moderate cartoon violence High cartoon violence # aggressive acts = 5 # aggressive acts = 11 # aggressive acts = 14 Page 12  Non-Systematic Variance Within-group (non-systematic) variance: any differences (variation) between subjects who are treated alike Participant Condition A Condition B Condition C 1 16 4 2 2 18 6 10 3 10 8 9 4 12 10 13 5 19 3 11 Examine scores of subjects in the same group: In Condition A, one participant has a score of “16”, another “18”, etc. In theory, they should be identical since they have been given the same treatment Differences can be due to chance factors, Experimental error Page 13 Testing Group Differences We use these two measures of variance: between-group and within-group variance, to form the condition or treatment index, on which the F-ratio is based F-ratio responsive to absence or presence of treatment effects: – i.e., the effect of our IV Page 14 Testing Group Differences How to compute Treatment index: Therefore, The treatment index is called the F-ratio Page 15 Testing Group Differences To determine statistical significance, you compare the between-group variance to the within-group variance F = between-group variance (systematic) within-group variance (error) When there is a large within-group variance, it is difficult to see an effect – Want to minimize it Large between groups variance is good Large within groups variance is bad Page 16 What a Between-Subjects F-ratio Represents Music Lessons No Lessons Measure of Between-group variance > Within-group Within-group variance variance F-ratio is Positive and Large Measure of Between-group variance Music Lessons No Lessons Between-group variance < Within-group variance F-ratio is near 0, Small Measure of Within-group variance Page 17  Between- and Within Group Variances in Charts Between-group variance > Change in IQ (points) 20 Within-group variance 18 16 14 12 10 8 6 4 2 Music Lessons No Lessons 0 Lessons No Lessons Between-group variance < Within-group variance 20 Change in IQ (points) 18 16 14 12 10 8 6 4 2 0 Music Lessons No Lessons Page 18 Lessons No Lessons  Testing Group Differences Small between-group variance or Large within-group variance are bad - will yield Small (non-significant) F-ratio Group 1 Group 2 Music Lessons SYSTEMATIC No Lessons VARIANCE NON-SYSTEMATIC NON-SYSTEMATIC 14 10 VARIANCE VARIANCE 12 11 18 9 16 7 15 10 Page 19  Comparing > Two Groups Single-factor multiple-group design Example: comparing driving performance under three telephone conditions Cell phone; handfree phone; no phone Analyzed with a single-factor analysis of variance (ANOVA) (between-subjects = independent measures) Provides stronger evidence for a real cause-and-effect relationship than a two-group design Page 20  Between-Groups Design Researchers try to maximize between-group differences – i.e., make sure you are comparing two distinct things 1 hour of violent cartoons vs. 1hr 15 min of violent cartoons Researchers try to minimize within-group differences – Make sure all participants within a group are treated exactly the same Standardize the experimental procedures – Minimize individual differences: Hold extraneous variables constant or restrict the range – Use a large sample size – Note: random assignment has no effect on within-group variance Page 21 Between-Groups Design Two major sources of confounding variables: Individual differences (from assignment bias) Environmental variables Page 22 Individual Differences Individual differences: any personal characteristic that differs from one participant to another (many!) – Age, sex, IQ, SES, personality, relationship status, health… Want to make sure that the different groups are as similar as possible, except for the IV Page 23 Age-Related Individual Differences Example: Study of drivers' memory for traffic patterns with / without cell phones With Cell phones Without Cell phones Page 24 Individual Differences This is called assignment bias: – When process of assigning participants produces groups with different characteristics Threat to internal validity Page 25 Environmental Variables Any characteristic in the environment that may differ – e.g., room, lighting, time of day, noises, presence of researcher, etc. If those differ between groups, we may have an extraneous variable that becomes a confound Can no longer say treatment (IV) caused outcome; could be environmental factor Page 26 Between Groups Design To establish equivalent groups of participants, they must be: - created equally - treated equally - composed of equivalent individuals Page 27 Limiting Individual Differences + Environmental Variables Randomization: – Participants are randomly assigned to groups to ensure groups are as equal as possible before treatment – Most powerful technique to control for the effect of pre- existing differences by equalizing them (spreading them evenly) Page 28 Limiting Individual Differences + Environmental Variables Randomization: Page 29 Limiting Individual Differences + Environmental Variables – Randomization is NOT the same thing as random sampling Random sampling = random selection of participant from a larger population to participate in the study Randomization = random assignment of participant to experimental or control groups in a particular study Page 30 Limiting Individual Differences + Environmental Variables Randomly assign to different conditions to ensure even distribution of individual differences across conditions (dice, table…) Page 31 Limiting Individual Differences + Environmental Variables Free Random Assignment: – Coin toss to ensure participants are assigned to groups solely on the basis of chance – Each P has an equal chance of being assigned to any one of the treatment conditions – Theoretically should lead to equality, but no guarantee Page 32 Limiting Individual Differences + Environmental Variables Improbable that groups perfectly matched, but will differ only randomly which is typically very small Therefore, differences among groups are treatment effects, and will neutralize nuisance effects at the same time With small samples, there are no guarantees Therefore, we can modify randomization to have control over the outcomes Page 33  Limiting Individual Differences + Environmental Variables Matching: Participants are matched on critical variables Examples: -Intelligence -gender -age -severity of illness Match subjects on pre-existing differences which may be related to differences in the DV: – Guarantees groups are equivalent on critical variables Page 34 Limiting Individual Differences + Environmental Variables Matching involves 4 steps: 1. Identify the variable(s) to be matched: – What (if any) are the potential confounding variables? 2. Measure and rank subjects on the variable for which control is desired (may require pretest) 3. Segregate subjects into matched pairs on that variable 4. Randomly assign pair-members to the conditions Page 35  Limiting Individual Differences + Environmental Variables Matching Example: effects of marijuana on pain relief in cancer patients who are in stage 1 (early) to stage 4 (late) onset Treatment No Treatment P1 = Stage 1 P9 = Stage 1 P2 = Stage 1 P10 = Stage 1 P3 = Stage 2 P11 = Stage 2 P4 = Stage 2 P12 = Stage 2 P5 = Stage 3 P13 = Stage 3 P6 = Stage 3 P14 = Stage 3 P7 = Stage 4 P15 = Stage 4 P8 = Stage 4 P16 = Stage 4 Page 36  Matching across Blocks Matching can be extended to units larger than pairs – Groups of individuals are matched in blocks IQ blocks: high group > 110 medium group = 90-110 low group < 90 Age blocks: pre-adolescent 9-11 adolescent 12-16 late adolescent 17-19 Page 37  Matching across Blocks Example: effectiveness of reading enrichment program on academic performance (matching on IQ) Treatment No Treatment P1 = 77 P10 = 79 P2 = 85 P11 = 75 P3 = 89 P12 = 80 P4 = 97 P13 = 99 P5 = 100 P14 = 102 P6 = 105 P15 = 95 P7 = 115 P16 = 117 P8 = 120 P17 = 124 P9 = 117 P18 = 128 Page 38  Threats to Internal Validity Attrition: – Refers to participants leaving the study prior to completion – Not a problem if members of all groups leave at same rate – Problem if they leave one group at a higher rate than in other groups (differential attrition) Groups are no longer equivalent Is difference between groups due to treatment effects or differential attrition? Page 39  Threats to Internal Validity Communication between groups: – Diffusion may occur Treatment effects spread from one condition to another condition – True effects of treatment may be masked by shared information One group is getting benefit of information from another group Appears to be no effect Page 40  Threats to Internal Validity Communication between groups can also result in: Resentful demoralization: – One group might receive course credit while another group receives payment - Any perceived inequity can influence behavior In these cases: – Observed differences between groups have alternate explanation Page 41 Advantages of Between-Subjects Design Simple design Each score is independent from other scores Clean and uncontaminated by other treatment factors: No carryover effects No practice effects, fatigue, boredom Experiment takes less time for each participant Causality can be established Page 42  Page 43 Why Between-Subjects Designs Are Popular Because carryover effects are of unknown duration: Therapy A Therapy B X = 1st-order carryover X Therapy A Therapy B Therapy C Y = 2nd-order carryover X X Y Therapy A Therapy B... Therapy N X X Z= Nth-order carryover Senn (1992) Z  Page 44 Why Between-Subjects Designs Are Popular One can estimate the bias introduced by carryover effects: Therapy A Therapy B... Therapy N X X Outcome Measure (%) Estimated Bias in 1st-order 1st,2nd,+3rd-order Senn (1992) Size of Carryover (%) Size of Carryover (%) Implication: Carryover effects create more bias as # Conditions increases Disadvantages of Between- Subjects Designs Requires many participants Each P contributes only one score: To compare, need to have n participants per group 30 in each group, 3 groups = 90 participants! Can be difficult to recruit enough participants in special populations ex: postpartum depression Individual differences and environmental differences can exist: Groups must be equivalent before the manipulation Generalization (external validity) can be hard if holding subjects constant on extraneous variables reduces their representation in the population Page 45  Disadvantages Assignment bias, experimenter-expectancy and subject-expectancy biases: Solutions: assign participants to conditions so that: a) participants are blind to (unknowledgeable about) the condition b) experimenters are blind to the condition c) both are blind (double-blind) Additional constraint that avoids analysis bias: d) Data analyst is blind to the condition of the participants Page 46  Page 47 Between-Subjects Design: When to Use Faked self-reports of health behavior: A comparison between a within- and a between-subjects design (Egele, Kiefer, & Stark, 2021). Participants were told to complete a health form: A) honestly, B) dishonestly but believably, or C) dishonestly and unbelievably In a between-subjects design they received only A or B or C to complete In a within-subjects design they received A, B, and C to complete Findings: Both designs resulted in differences between conditions Respondents in the within-subjects design showed larger differences between the 3 conditions Implications: Within-subjects designs raise participants' perception of differences across conditions (Comparison effect) Use Between-subjects design to avoid carryover / comparison effects Between-Subjects Design: When Not to Use How to show that 9 > 221: Collect judgments in a between-subjects design (Birnbaum, 1999) Between-subjects: "How large is the number 221?" or "How large is the number 9?" 40 raters/group used a 10-point scale (1= very very small; 10=very very large) One group rated "9": Mean Rating = 5.13 ! Another group rated "221": Mean Rating = 3.10 ! Reasons: The groups can use different "anchors" on the rating scale when each participant completes only one condition Do not use Between-subjects design when similar anchoring is desired across conditions Page 48 Between-subject designs avoid some threats to validity Important considerations - does the between-subjects design avoid: Carry-over effects (influence of one condition on other conditions) Participant awareness (sensitized to task measures over time) Ecological validity (if participant is not usually exposed to all levels of a variable in nature, then results not representative) Changes in measurement properties / tests over time Montoya (2022) Page 49  NEXT TIME: CHAPTER 9 WITHIN-SUBJECTS DESIGNS Page 50