McGill Research Methods in Psychology Past Paper PDF

Research Methods in Psychology Chapter 7: The Experimental Research Strategy Page 1  Midterm 1 Grades Grades will be posted online by Sunday Oct 27 on MyCourses under "Assignments". one total mark for multiple choic...

Research Methods in Psychology Chapter 7: The Experimental Research Strategy Page 1  Midterm 1 Grades Grades will be posted online by Sunday Oct 27 on MyCourses under "Assignments". one total mark for multiple choice out of 28 points (we removed 2 questions) one total mark for short answers out of 4 points FINAL GRADE: out of 32 Answers to the short-answer questions will be posted as well. Summary statistics: Average multiple choice grade = 24 / 28 (range = 16 - 32) Average short answer grade = 3 / 4 (range = 0 - 4) Overall average = 27 / 32 Exams will not be returned; students who scored 22 / 28 or lower on multiple choice or 2/4 or lower on short answers can view their exam at office hours by Oct 31 2  Questions Removed (Psychometric Analysis) A researcher wants to select a sample of 40 people so that four different religious groups are equally represented. Which sampling technique should they use? a. Quota sampling b. Stratified random sampling c. Proportionate stratified random sampling d. Convenience sampling A scientist developed a new test of anxiety. People scored high on the new measure as well as on a standardized test of anxiety. The new test demonstrated high ____. a. concurrent validity b. divergent validity c. reliability d. predictive validity 3  Midterm 1 Grades Who/what/where for grading information MULTIPLE CHOICE (MC): Students who received a mark of 22/28 or lower will have until end of day Thursday Oct 31 to attend an office hour to ask about their marks Last name begins with A-L: see TA Charlotte Last name begins with M-Z: see TA Rachael Please - do not visit the office hours if you scored 23/28 or higher on the multiple choice 4  Midterm 1 Grades Who/what/where for grading information SHORT ANSWER (SA): Students who received a mark of 2 or lower will have until end of day Thursday Oct 31 to attend an office hour to ask about their marks All students: visit Dr. Palmer's office hours Please - do not visit the office hours if you scored 3 or higher on the short answers If you cannot make these office hours: - see another TA for multiple choice grades - write Dr Palmer for short answer grades If time permits: students with 23 / 28 multiple choice or 3 / 4 short answer can visit office hours about grades on the week of Nov 4-7 5  Scoring of Short Answers 0 = all incorrect or missing; 1 = part correct, part incorrect; 2 = all correct, nothing missing) 1. Name and describe two important historical events discussed in this course that led to the development of psychologists' ethical guidelines for conducting research. Tuskegee syphilis study in which blood draws taken from black men who were not informed of the study purposes or offered treatment when it became available. Nazi atrocities of scientists putting humans at risk or death, which came out in the Nuremburg trials and Nuremburg code. Milgram's experiments on obedience which put participants at risk of guilt and shame through deception about shocking other participants, without proper debriefing. Nootka (BC) study of arthritis which used genetic samples of indigenous tribe to study other diseases without consent. Incorrect Examples (incomplete list): Naming ethical guidelines (such as Belmont) without describing the historical event that led to its development; naming but not describing. 6  Scoring of Short Answers 0 = all incorrect or missing; 1 = part correct, part incorrect; 2 = all correct, nothing missing) 2. A scientist is studying the transportation methods used by Montreal residents by recruiting participants from bus stops and sidewalks for a one-month summer period. Name and describe two potential sources of sampling bias in this study. One source: choice of locations to sample is limited, which omits other popular transportation methods (such as car, metro, bicycle, etc). Another source: choice of timepoints (one season) to sample, which can bias toward outdoor transportation methods that are used less than other (indoor) methods in other seasons (metro, car, etc) or used less by some populations (students) who are not present in summer months. Examples of Incorrect answers (incomplete list): naming types of sampling biases (convenience, snowball, volunteer) that are unconnected to the specific study described (failure to address "in this study" portion of question). 7  Scoring of Short Answers 0 = all incorrect or missing; 1 = part correct, part incorrect; 2 = all correct, nothing missing) 3. A researcher wants to know if smoking causes decreased sleep quality. Heavy (frequent) and light (infrequent) smokers were recruited; the researcher measured the number of cigarettes smoked per day and nightly sleep quality. Name and describe one possible confounding variable and explain why it is a confound and not an extraneous variable. Caffeine/coffee/alcohol intake; known to disrupt sleep and known to be correlated with smoking. A confound because the variable affects both IV and DV; an extraneous variable affects IV or DV but not both. Stress /anxiety/ depression: known to disrupt sleep and can cause smokers to smoke more. It is a confound because it affects both IV and DV; an extraneous variables affects IV or DV but not both. Hormone production (affects sleep and smoking). Income: can afford cigarettes, better sleep conditions (beds). Answer must include explanation of why proposed variable is a confound, including description of its effects on IV and DV, as 8 well as definition of extraneous variable.  Scoring of Short Answers 0 = all incorrect or missing; 1 = part correct, part incorrect; 2 = all correct, nothing missing) 3. A researcher wants to know if smoking causes decreased sleep quality. Heavy (frequent) and light (infrequent) smokers were recruited; the researcher measured the number of cigarettes smoked per day and nightly sleep quality. Name and describe one possible confounding variable and explain why it is a confound and not an extraneous variable. Examples of Incorrect answers (incomplete list): Noise in neighborhood (known to affect sleep but not known to affect smoking). Time spent exercising/pre-existing health (known to affect sleep but not known to affect smoking rate). SES (includes education, wealth, status; need to define which variable). Sleep supplements (medication): known to affect sleep but not smoking rate. Emotional states of participants. Sleep health. Smoking history. Working hours per day. Missing explanation for why variable is not an extraneous variable. 9 Outline - Experimental Designs Introduction: – Problems with causality, experimental control and experimental groups Unique attributes of the experimental design: – Manipulation and control External validity: – Simulations and field studies 10 Experimental Strategy Most powerful of the research methods – Contains high constraints on variables The main goal of an experiment is to arrive at a causal explanation – Done through experimental design Manipulation of one variable (IV) to demonstrate its effect on another variable (DV) while holding other potential influences constant 11  Basic Goal of Experimental Design Demonstrating a cause-and-effect relationship: 1) Establish that the “effect” happens after the “cause” occurs Changing the value of the IV is followed by a change in the DV. 2) Establish that one specific variable (IV) is responsible for changes in another variable (DV) An experiment must rule out the possibility that changes are caused by an extraneous variable. Page 12  Experimental Strategy Strive to provide a comparison of situations (conditions) in which the proposed cause is present or absent Absent Present 13 Experimental Strategy In order to establish causality, we use strict control – essentially creating an unnatural situation where variables are isolated from influence of other variables Only Experimental or Quasi-experimental designs can establish causality. It is not possible to prove causality, but only to show its likelihood relative to chance (alpha value associated with statistical test) 14  Establishing Causality To establish whether two variables are causally related: whether a change in the IV (X) results in a change in the DV (Y), one must establish: 1) time order-The cause (X) must have occurred before the effect (Y) 2) co-variation (statistical association)- Changes in the value of the IV (X) must be accompanied by changes in the value of the DV (Y) 3) rationale/explanation- There must be a logical and compelling explanation for why these two variables are related 4) non-spuriousness- It must be established that the IV (X), and only the IV, caused the observed changes in the DV (Y); rival explanations must be ruled out. 15  Establishing Causality 1) Time order-The cause (X) must have occurred before the effect (Y) Example: Individuals arrested for domestic assault tend to commit fewer subsequent assaults than similar individuals who are accused in the same circumstances but are not arrested. Cause Effect Adults accused of Those arrested -> Fewer assaults later domestic violence: Those not arrested -> More assaults later Sherman & Berk (1984) Motivational speakers: You must believe in yourself BEFORE you can achieve success. 16 Often, those who achieve success THEN believe in themselves. The temporal order of self-belief and success is unknown.  Establishing Causality 2) Co-variation- there is an empirical association between the IV (X) and the DV (Y) that results in co-variation. Examples in which the two variables vary together: The more years of schooling, the higher the projected income will be later. The more cigarette smoking, the greater the odds of lung cancer. Example in which the two variables do not covary: The use of the death penalty in US does not lead to reduced violent crime. There cannot be a cause-and-effect relationship without an association between those variables. 17  Establishing Causality 3) Rationale/explanation: a compelling explanation for the connection between the IV (X) and the DV (Y). What is the mechanism that accounts for why the IV influences the DV? Example: relationship between childhood poverty and petty crime (theft, trespassing, etc) Proposed rationales: low parent/child attachment, inadequate child supervision, erratic discipline Sampson & Laub, 1994 Figuring out the process by which the IV influenced the DV can increase confidence in a conclusion that there was a causal effect 18  Establishing Causality 4) Non-spuriousness: (spurious = false or not genuine) A relationship between two variables is spurious if the relationship is actually due to a third variable. What appears to be a direct connection between X and Y is indirect. Example: Grade schools with larger libraries cause or produce students who are better readers. Possible third variable: parents' education/views on education. These parents may be more likely to have more books, read to their children, use a library. Parents’ education may cause variation in school libraries and students' reading level. If so, the association between school libraries and student performance 19 would be at least partially spurious. The Experimental Strategy Two basic types: Between Groups Within Groups 20 Between-Groups Design Two or more samples (groups) are formed at random from a pool of subjects Each group is composed of different participants Each group is assigned to a different condition (value of the IV) and the values of each group are compared Cartoons with Violence Cartoons with DV: aggressive behaviors Compare Mean of Group1 no violence with Mean of Group2 21  Within-Groups Design Only one sample (group) is formed and each person participates in all conditions (levels of the IV) Values are compared across conditions within participant (values from the same participant in different conditions) Cartoons with Violence (Condition1) A single group of Participants Cartoons with no violence (Condition2) DV: aggressive behaviors Compare Difference in each Participant’s 22 Condition1-Condition2 means Independent / Dependent Variables Independent variable: – The variable we think is the cause – The manipulated variable in experimental studies Is manipulated by creating a set of treatment conditions Dependent variable: – The variable we think is the effect – Results from the manipulation of the IV (cause) – The outcome – what is measured in each of the conditions Extraneous variables: – Any other variable in the study 23 Between-Group Designs: Experimental Group: – The participants in your experiment exposed to an experimental manipulation Control Group: – A group in your experiment that is not exposed to the manipulation and that is used for comparison purposes – Example: Treatment vs. placebo 24  Experimental Conditions Does children’s exposure to music lessons (IV) improve their intelligence scores (DV)? IV = Lessons Lessons= Between-Subjects IV Music Lessons (treatment) No lessons (control) Time of Testing= Within-Subjects DV = Change in Intelligence test scores IV from Before-lessons to After-lessons Answer: yes, but slightly (Schellenberg, 2011) 25 Possible Control Groups 1) Placebo control group: Ps receive a fake (placebo) treatment, with no real treatment Placebo effect: Just believing that the treatment will have an effect can cause a response Treatment / Placebo comparisons indicate effect of treatment on DV beyond placebo effects 2) Waitlisted group: Participants who sign up to receive same treatment Controls for motivation across groups Comes at a cost: experimenter must offer tx 26 Experimental Conditions Is the antidepressant Prozac (IV) an effective treatment for depressive symptoms (DV)? IV = Antidepressant Treatment (Prozac) Placebo (control) DV = Change in depressive symptoms = After Prozac minus Before Prozac 27 Importance: Ebola Example Nurse's aide in Spain recovered from Ebola: “She has received two main treatments in her battle against Ebola, for which there is still no vaccine" (vaccine developed 2019) First treatment = anIV drip with the antibodies of anEbola survivor. Second treatment = an experimental anti-viral drug, favipiravir How do we know what treatment worked? 28  The Experimental Research Strategy Four basic elements: 1. Manipulation Researcher manipulates one variable by changing its value to create conditions. 2. Measurement A second variable is measured for each condition, resulting in a set of scores in each treatment condition. 3. Comparison The scores in each condition are compared with the scores in other conditions. 4. Control All other variables are controlled to be sure that they do not influence the variables being examined. 29 Four Basic Elements Control other Page 30 variables The Experimental Research Strategy 2 of these 4 elements are unique to experimental research: Manipulation Change in IV was Measurement followed by change Comparison in DV Rule out (control for) Control other possible variables 31 1. Manipulation First decide on levels of IV you would like to examine – TV (cartoon) violence? No violence 5 violent acts in 15 violent acts in 15 min. cartoon 15 min. cartoon None Small amount Large amount 32 1. Manipulation First decide on levels of IV you would like to examine – Prozac application for depression? None 10 mg Prozac 20 mg Prozac None Small amount Large amount 33 1. Manipulation & Directionality Primary purpose of manipulation is to determine the direction of the effect Manipulate one variable (cause it to increase/decrease) and observe second variable: – Is it affected by this manipulation? TV Aggressiveness Violence Causality in relationship between IV + DV assumes researcher has controlled all other variables 34 1. Manipulation & Directionality Primary purpose of manipulation is to determine the direction of the effect Manipulate one variable (cause it to increase/decrease) and observe second variable: – Is it affected by this manipulation? Ice cream Crime rate consumption Causality in relationship between IV + DV assumes researcher has controlled all other variables 35 1. Manipulation & the Third Variable Problem Manipulation can also be used to control the influence of outside variables. Temperature Ice cream Crime consumption rate Page 36 1. Manipulation & The Third Variable Problem Active manipulation of IV allows us to ensure that the IV is not changing along with another variable that can account for the relationship you are interested in Being Suicide Bullied Attempts Self- Esteem Researcher is responsible for causing the IV to change – Can be confident that changes in IV not caused by some outside variable – but only if third variables are controlled for 37  Manipulation 2. Control Measurement Comparison Control Must make sure there are no other factors contributing to the changes observed in the DV beyond that of the IV Must rule out all other possible explanations for changes in DV (eliminate confounding variables that can influence both the IV and the DV) 38 2. Control Extraneous Variables: – All other “extra” variables that are present in the study but are not studied or investigated – They do not matter if they do not affect the outcome and do not correlate with the manipulated variables – Examples: age, gender, ethnicity, personality, environmental variables 39 2. Control Confounding Variables: – A special class of extraneous variable that changes systematically with an IV and can influence the DV – therefore can mask the true effect of the IV – They matter because they vary systematically along with the IV and can affect the outcome – When IV and confounding variables change together systematically, cannot conclude cause/effect between IV/DV Provides alternative explanation of results 40  2. Control Confounding Variables: Hypothesis: Sweetened cereals are preferred more than unsweetened cereals Confounding: the sweetness AND the color vary together in the IV. And the color varies in the levels of the DV. Page 41  2. Control Eliminating the Confound: Hypothesis: Sweetened cereals are preferred over unsweetened cereals Control the color Page 42 2. Control Example: Does exposure to music lessons (IV) improve IQ test performance (DV)? Experimental group Control group Music+ No music, teacher No teacher Higher IQ score Lower IQ score Is exposure to music (IV) affecting IQ performance (DV)? Or is it the presence of the teacher - possible confound? 43 2. Control Extraneous vs. Confounding Variables You are conducting a word memory experiment with 2 conditions (background music present / absent) and there is noise due to construction around the lab Noise would be an extraneous variable if it affects all conditions (same for all participants) If only one condition is affected and there is reason to believe noise will impact the memory DV, it is a confounding variable because the effect of the noise on memory becomes confused with the effect of the IV (background music present/absent) on memory 44 2. Control Example: Does caffeine intake affect subsequent memory for new material? Experimental group Control group Caffeinated drink Decaffeinated drink Noise Noise Memory score Memory score Only difference is the IV (caffeine) Noise = extraneous variable (not a confounding variable) 45 2. Control Confounding variables (CV) are an important threat to internal validity: Do not know if it is IV or confounding variable (CV) that is causing the change in DV The presence of confounding variable offers an alternative explanation for your results - Introduces ambiguity Music Lessons (IV) IQ performance (DV) Teacher Presence (CV) Page 46 2. Control Confounding variables are an important threat to internal validity: Do not know if it is IV or confounding variable (CV) that is causing the change in DV The presence of confounding variable offers an alternative explanation for your results - Introduces ambiguity Sleep (IV) Memory score (DV) Noise (CV) Page 47 2. Control Less obvious example (Schmidt, 1994): – Examined effects of humour on memory – Used humorous and nonhumorous sentences with same meaning Individuals recalled more humorous sentences than nonhumorous. Is there a confounding variable? Page 48  2. Control Beneficial effects of humour on memory due to surprise Surprise humor caused participants to pay more attention to the humorous sentences – Level of surprise varied systematically with humour Open question: would any surprising stimuli improve memory? 49  2. Control What makes an extraneous variable become a confounding variable? Two important characteristics: 1) The variable must affect the DV We do not care about variables that will not influence the DV (they are not a threat) AND 2) The variable must vary systematically with the IV We do not care about variables that change randomly with the IV (they are not a threat) A variable that affects the DV with no relation to the independent variable is not a threat 50 2. Control How do we deal with confounding variables? Examine extraneous variables for possible influence on DV and on IV Three categories of extraneous variables: Environmental variables (different testing environments, time of day) Participant variables (individual differences such as gender, age, personality, IQ, etc.) Time-related variables (tested over time; history, maturation, instrumentation, testing effects, regression to the mean) Consider these categories for possible confounds 51 Research Methods in Psychology Chapter 7: The Experimental Research Strategy, ctnd Reminder: HW3 posted now under Quizzes Read paper by Hyman 2009, under Content/Readings Due Nov 5 by 9pm 52 2. Control – Confounding variables We need to introduce experimental control so that extraneous variables do not become confounding variables Five ways to control for possible confounding variables: A. Remove them B. Hold them constant C. Use a placebo control D. Match them across conditions E. Randomize them 53 A. Remove the Confound How can we deal with confounding variables? We can eliminate them… – e.g., take away element of surprise, change testing room, etc… But we cannot eliminate all confounding variables… 54 A. Remove the Confound How can we deal with confounding variables? Experimental group Control group Caffeinated drink Decaffeinated drink No Noise No noise Memory score Memory score We can eliminate them… – change the testing room, etc… But we cannot eliminate all confounding variables… 55 B. Hold the Confound Constant If we cannot remove the confounded variable, we can try to hold it constant across conditions Independent + Confounded variable variable T Dependent Confounded variable Variable C Examples: Caffeine, surprise, music teacher Holding a variable constant eliminates its potential to become a confound 56 B. Hold the Confound Constant By standardizing the environment and procedures (no noise, presence of music teacher), most environmental variables are held constant Independent + Confounded variable variable T Dependent Confounded variable Variable C 57 B. Hold the Confound Constant Can also standardize the confound to a range of values –example: age and gender standardized (only 30-35-year-old females) Problems: Too strict control can be unreasonable Usually have a range participants between 18-21 with IQs between 100 and 110 All groups roughly equivalent Trade-off between standardizing and external validity 58 Cannot generalize beyond this sample C. Use a Placebo Control Sometimes the experimental method itself can become a confounding variable – To control for this we use a placebo control group Effects of confounding variable + Treatment Dependent Variable Minus Effects of confounding variable alone Dependent Variable 59 C. Use a Placebo Control Sometimes the experimental method itself can become a confounding variable – To control for this we use a placebo control group Delivery method = source of anxiety Treatment: Vaccine + Placebo Control: Water + 60 D. Matching Across Conditions If we cannot remove or hold the confounded variable constant, we can try to match levels of it across conditions (balanced) Matching levels of variable Age Matched Location Matched Experimental Control Experimental Control 20 20 Room1 Room1 20 20 Room1 Room1 20 20 Room1 Room1 25 25 Room2 Room2 25 25 Room2 Room2 25 25 Room2 Room2 But matching based on fixed values can limit the generalizability (threat to external validity) 61 D. Matching Across Conditions If we cannot remove or hold the confounded variable constant, we can try to match average levels of it across conditions (balanced) We can use counterbalancing of variables to reduce effects due to different average values Matching average values Average Age Matched Location Matched- Counterbalanced Experimental Control Experimental (C1) Control (C2) 23 26 1 2 21 25 1 2 26 24 1 2 24 22 2 1 25 26 2 1 27 21 2 1 When averages are used, counterbalancing of other factors can be beneficial 62 D. Matching Across Conditions Problems: Much time and effort Reduced participant sample to choose from Can we control for/match on all extraneous variables? Recommended for specific set of variables that pose serious threats to internal validity 63 E. Randomizing Participants assigned to Conditions Randomly assign participants to treatment conditions so that the extraneous variables related to participants will balance out across the conditions The aim is to disrupt any systematic relationship between the extraneous and independent variables (to prevent the EV from becoming a CV) Powerful method for controlling many environmental and participant variables simultaneously rather than individually 64  E. Randomizing Participants assigned to Conditions Example: Participants recruited for experiment on one of 3 testing days Each participant assigned randomly to "Intervention" or "Control" conditions Assign each participant randomly as they appear on testing day Testing Day 1 Testing Day 2 Testing Day 3 Page 65 E. Randomizing Participants assigned to Conditions Uses unpredictable and unbiased procedure to distribute different extraneous variables randomly across different conditions Random assignment to treatment conditions: – Use a coin toss to assign each participant – Assignment is random, therefore extraneous variables cannot vary systematically between groups Page 66  E. Randomizing Participants assigned to Conditions Does not guarantee control: Uses chance factors which can produce a biased outcome – still possible that all people with similar backgrounds (potential CVs) are assigned to one condition With large enough numbers, randomizing guarantees a balanced result (Groups of >=20 participants per condition) Music None Example: effect of music lessons/ Testing Day 1 Lessons No music lessons on IQ scores Testing Day 2 Confound: those assigned to Music lessons happened to have more prior music exposure Testing Day 3 Page 67 Manipulation Checks Did your manipulation have the intended effect? Were participants aware of the manipulation in your experiment? How did they perceive and interpret it? 1. Check the manipulation: take measures of the IV to make sure your manipulation did what you wanted to do (e.g., sad vs. happy mood) 2. Include an exit questionnaire that tests whether the participants were aware of the manipulation(s) and purpose of the experiment ("what did you think the purpose of this experiment was? Did you notice…?") 68 Manipulation Checks Important to do when: 1. Participant manipulations: difficult to know if worked (especially compared to environmental manipulations) – Include a measure of IV (e.g., mood, frustration, stress) to assess if worked 69 Manipulation Checks Important when: 2. Subtle manipulations: difficult to know if participants noticed – Exit questionnaire e.g., “Did you notice the expression on the experimenter’s face when she gave you the instructions?” 70 Manipulation Checks Important when: 3. Placebo controls: did participants believe placebo was real? – Exit questionnaire e.g., “What treatment did you receive? Did you feel it was effective? Were you aware you were being deceived about X?” 71 Manipulation Checks Important when: 4. Simulations: difficult to know if participant perceives environment as real – Exit questionnaire e.g., “What did you think when the other participants answered incorrectly? To what extent did you think about the fact that you were in an experiment?” 72  Possible Reasons that an Experiment did not work Dessert Reason Example Possible solution Preferences IV is not sensitive IV = 2 food items to test Include more foods % Yes responses 100 enough preferences: chocolate or 80 beets 60 40 DV is not sensitive 2 levels of preference to rate: Use rating scale (7- 20 enough "yes" or "no" point) 0 IV has floor or Chocolate IV is at ceiling; Include test items Cookies Chocolate ceiling effect beets IV is at floor not as preferred/avoided DV has floor or "yes" response is at ceiling Include responses in Food Preferences ceiling effect and "no" is at floor middle range Measurement Subject variables (hunger); Control those % Yes responses 100 80 error noisy test environment variables 60 Insufficient power Not enough participants to Increase sample size 40 20 detect a true effect of IV 0 Hypothesis is People do not prefer specific Compare with Beets Chocolate wrong! food previous studies 73 Food Type  8 Threats to Internal Validity 1) History: did some other current event effect the change in the dependent variable? Did all groups experience the same current events? Example: prominent hiphop star arrested on drug charges; fans affected differently than non-fans in how they complete survey on interest that day about music videos 2) Maturation: were changes in the dependent variable due to normal developmental processes? Example: children age / change more quickly than adults 3) Statistical Regression: did subjects come from low or high performing groups that will naturally generate scores more toward the mean? Example: Two schools compared pre-, post- test reading scores after 4-week reading program; Differences within each school may arise because the low scores (more in school 1) cannot go lower. 74  8 Threats to Internal Validity 4) Selection: were the participants self-selected or assigned randomly? Example: adolescents who choose to see a therapist may not be representative of the population 5) Experimental Attrition: Did some participants drop out in unequal numbers across conditions? Example: students who most dislike college drop out, and are not included in a survey of whether study centers were better in high schools or universities. 6) Testing: Did previous testing affect the behavior at later testing? Example: administering different vaccines to same participants over time reduces their needle anxiety; last vaccine may be most effective due to testing effects 75  8 Threats to Internal Validity 7) Instrumentation: Did the measurement method change during the research? Example: Interviewer gets tired, does not ask as many questions of last participants in study with 2-hour interview. 8) Design contamination: did participants find out something about the experimental conditions? Example: People in a 2-person study about decision-making notice there is less opportunity to win in one condition than another; they start to compete more, believing that they are being judged 76  4 Threats to External Validity 1) Unique program features: There may have been an unusually motivated set of experimenters in some conditions. Can this experiment be replicated in another lab? 2) Effects of Selection: Was the recruitment and assignment of participants to conditions successful? Can this study be replicated with different participants? 3) Effects of Environment / Setting: Can these results be replicated in other labs or other environments? 4) Effects of History: Can these results be replicated in different eras / time periods? 77  External Validity & Experimental Strategies Lab simulations: – Trying to bring the “real world” into the laboratory to increase external validity – By creating conditions within an experiment that closely duplicate the natural environment 78  External Validity & Experimental Strategies Lab simulations: – Trying to bring the “real world” into the laboratory to increase external validity – By creating conditions within an experiment that closely duplicate the natural environment Mundane realism: How close the lab environment is to the real world Experimental realism (what researchers aim for): Bringing only the psychological aspects into the lab (participants immersed in simulation may behave normally, not thinking or remembering they are in experiment) 79  External Validity & Experimental Strategies How well did original lab simulations work? – Examination of published simulations for psychological issues (pre-virtual reality) Hammerl, 2000 – Strong preference for hypothetical scenarios as independent variables (manipulations) – Strong preference for qualitative self-reports as dependent variables 80  External Validity & Experimental Strategies How well do current-day lab simulations work? Current-day simulations: Realistic immersive stimuli (independent variables) Quantitative response measures (dependent variables) Newman et al (2022) 81  External Validity & Experimental Strategies How well do current-day lab simulations work? High realism virtual reality (VR) environments influence viewers’ emotional responses More positive affect when immersed in nature Similar to responses observed in actual nature = Experimental realism Newman et al (2022) 82  External Validity & Experimental Strategies Field studies: – Trying to bring the experimental strategy into the real world to increase external validity – Can examine behaviors that would be difficult to simulate in the lab Example: bystander effects Kitty Genovese attacked in park (2014) 38 neighbors witnessed attack; no one intervened or called police. "Diffusion of responsibility" Bystander effects may differ in a public area (bus, metro) than in lab Page 83  External Validity & Experimental Strategies Field studies: – Trying to bring the experimental strategy into the real world to increase external validity – Can examine behaviors that would be difficult to simulate in the lab Field study (HW3): Study of attentional blindness for unexpected objects Clown cycled through campus Experiments measured observers’ awareness Hyman et al (2001) Page 84  External Validity & Experimental Strategies Field studies with animals: – Trying to bring the experimental strategy into the real world to increase external validity – Can examine behaviors that would be difficult to simulate in the lab Spatial cognition in homing pigeons: Taught a rooftop loft location Then birds released 1 – 18km away Wilkie et al (1989) Page 85  External Validity & Experimental Strategies Strengths: Both simulations and field studies allow researchers to test behavior in more realistic environments than laboratories Weaknesses: Field studies are difficult venues for controlling all extraneous factors Simulations are dependent on whether the participant believes the simulation is real (ie, performs the same as in natural settings) 86  Perils for Experimental Designs 1) Although experimental research requires theories for framing hypotheses for testing, some current experimental research is atheoretical. Without theories, the hypotheses tend to be ad hoc (after the finding) and possibly illogical or meaningless. 2) Many measurement instruments (questionnaires, rating scales, equipment) used in experimental research are not tested for their reliability and validity and can be incomparable across studies. Consequently, results generated using those instruments are also incomparable. 87  Perils for Experimental Designs 3) Experimental research sometimes uses inappropriate research designs, such as irrelevant dependent variables, no tests for interactions, no experimental controls, and non-equivalent stimuli across conditions. Findings from such studies tend to lack internal validity. 4) The conditions used in experimental research may be incomparable or inconsistent across studies. The use of inappropriate tasks for participants introduces threats to external validity (would other participants have responded differently), making comparison of findings across studies difficult. 88  Ways to Avoid the Perils: - Use pre-validated stimulus materials and tasks if available - Conduct treatment manipulation checks (by debriefing subjects after performing the assigned task) - Conduct pilot tests with a small sample to ensure the roles of the IV and DV - Use tasks that are simpler and familiar for the participants, instead of tasks that are complex or unfamiliar 89 NEXT TIME: CHAPTERS 8 & 9 BETWEEN & WITHIN SUBJECTS DESIGNS Page 90