ITRIP Lecture 5 - Experimental Research Design 2024 PDF

SCIENTIFIC FOUNDATIONS OF PSYCHOLOGICAL SCIENCE ITRIP Lecture 5: Experimental Research Design Dr Melanie Murphy [email protected] Reading: Navarro DJ and Foxcroft DR (2022). learning statistics with jamovi: a tutorial for psychology students and other beginners. (Version 0.75). Section 2.6 - A...

SCIENTIFIC FOUNDATIONS OF PSYCHOLOGICAL SCIENCE ITRIP Lecture 5: Experimental Research Design Dr Melanie Murphy [email protected] Reading: Navarro DJ and Foxcroft DR (2022). learning statistics with jamovi: a tutorial for psychology students and other beginners. (Version 0.75). Section 2.6 - Assessing the validity of a study (last week) Section 2.5 – Experimental and non-experimental research Optional Reding: Howitt, D., & Cramer, D (2011). Introduction to research methods in psychology. Pearson/Prentice Hall. (pp. 266-279) - Chapter 9 – The basic laboratory experiment Field (2017) Chapter 1, Sections 1.6.5 WHAT HAVE WE COVERED SO FAR? Definition of concepts is important for scientific communication. Constructs are concepts operationalized. More precise operationalisation allows for more precise measurement. Different scales of measurement provide different amounts of information Observable, quantifiable Nominal, Ordinal, Interval, Ratio Considering Reliability and Validity help us make judgements about the quality of a study design WHICH TYPE OF RELIABILITY? Match definition of reliability with its term… Which term?…… Stable measurement result over time? Cronbach’s alpha Different people administer the measure and get the same result? Inter-rater reliability Two versions of the measure produce the same result? Test-retest reliability Scores from one half of the measure match (correlate) with the other half? Split-half reliability The relationship between different combinations of the measure items match? Parallel forms reliability IMPROVING RELIABILITY Improving Improving measurement instrument increasing number of items, clearly written items, objective scoring technique Standardising Standardising test situation clear instructions, standardised setting Stating Stating the limitations and/or adjusting for them e.g., age norms, gender norms, education norms ! EXPERIMENTAL RESEARCH Method of investigation determines the internal and external validity of the study Use of a control group and randomisation of participants to treatments are hallmarks of a true experiment Elimination (‘holding constant’), or minimisation, of other possible explanations of the effect. If this is not possible, distribute (‘balancing’) these influences between groups The Research Process 1 Formulating a theoretical rationale 2 Designing the method of enquiry 3 Analysing the data 4 Interpreting the results X Y FINDING AN EFFECT Testing a hypothesis Is there an effect of the independent variable on the dependent variable? Will people in different groups show different outcomes based on a certain measure? Everyone gets measured with the same thing, how they differ will be based on the characteristics or experience of their group (condition). The ability to see this effect will be determined by our operationalization of the experiment How do we define our group characteristics? What is our manipulation? How do we measure change (reliably)? Trusting the results Can we trust our findings based on what we did to control potential confounds (Internal validity) Can we generalize our findings to other people and other situations that have similar characteristics with similar measures (External validity). The Research Report 1 Introduction, aims, hypothesis (X 2 Method Participants (who?) Measures (what X?, what Y?) Design and Procedure (how?) 3 Results 4 Discussion Y) Maximising hypothesized systematic variance (i.e., effect of X) FEATURES OF A ‘GOOD’ EXPERIMENT Eliminating extraneous systematic variance or balancing them equally between groups (i.e., high degree of control of variables in the experiment to exclude rival explanations) Minimising non-systematic (error) variance (i.e., measurement errors, individual differences) WHAT IS A GOOD DESIGN? Christensen, L., Johnson, R., Johnson, R. B., Turner, L., Christensen, L., Johnson, R., Johnson, R. B., & Turner, L. (2015). Research methods, design, and analysis ebook pdf, global edition. Pearson Education, Limited. (p. 237) Hypothesis: EXAMPLE STUDY: KNOWLEDGE REDUCES PREJUDICE Increasing knowledge of the group of people will lead to a reduction in prejudice towards them Independent variable (knowledge) film about asylum seekers (e.g., Cast from the Storm) Dependent variable (attitude) prejudice towards asylum seekers http://www.abc.net.au/tv/programs/cast-from-the-storm/ Test Yourself In research terms, this type of study is called a oneWhat are the disadvantages of p shot case study design, as shown in the following table. For this design and the rest that follow, we’re showing you events that occur in a sequence such as a group of True Experimental De participants being assigned to a group and then some kind of treatment being administered and then some posttest is True experimental designs includ (one-group post-test design) given (in this example). and assigning subjects in a rand Pre-Experimental Design: A one-shot study Step 1 Step 2 Step 3 Participants are assigned to one group A treatment is administered A posttest is administered A group is exposed to some type of treatment and then tested. What shortcomings might you notice about this one-shot case study type of pre-experimental design? First, no attempt at randomization has been made. How might this one-shot case study be used? It would not be very useful for experimental work or for establishing cause-andeffect relationships, but it would be acceptable if you were speculating about factors that occurred at an earlier time and the effect they had on later behavior. Another pre-experimental design, called the one-group pretest posttest design, is represented by the following: Film test group, thereby lending a strong and-effect relationship. One of t are so powerful is that they all of participants, random assign random assignment to groups. True experimental designs control sele to groups, and assignment of treatme For example, let’s look at on Method: Show the film to a sample of university these designs, the pretest postte students and then test them with a questionnaire onthis: which looks like prejudice. Step 1 Step 2 Random assignment of participants to a control group A pretest is administered Results: Prejudice scores were lower than the ‘norm’. Step 1 Step 2 Step 3 Step 4 Participants are assigned to one group A pretest is administered A treatment is administered A posttest is administered Fig. Random assignment A pretest is of participants to administered the experimental (or 185 Salkind (2018). Pg treatment) group(s) PREEXPERIMENTAL DESIGN: PROBLEM Problem: University students might already have lower prejudice scores than the ‘norm’ (before seeing the film). This selection bias (of the sample) affects the internal validity of the study. this one-shot case study be used? It would not be very useful for experimental work or for establishing cause-andeffect relationships, but it would be acceptable if you were speculating about factors that occurred at an earlier time and the effect they had on later behavior. Another pre-experimental design, called the one-group pretest posttest design, is represented by the following: For example these designs, th which looks like Pre-Experimental Design: Step 1 A one-group pre-test post-test design Random assignment Step 1 Step 2 Step 3 Step 4 Participants are assigned to one group A pretest is administered A treatment is administered A posttest is administered For example, a researcher is interested in studying how effective method A is in increasing muscle strength. The researcher follows these steps in the completion of the experiment: Pretest Film of participants to a control group Random assignment of participants to the experimental (or treatment) group(s) Post-test1. For this desig Randomly a group or the Method: Pre-test a sample of university students, show the each g 2. Pretest 1. Advertises for volunteers for the experiment film, and then test them again with a questionnaire 3. onApply the tr control group prejudice. 2. Administers a pretest to measure each participant’s muscle strength Results: 4. Posttest both 3. Exposes the participants to the hypothesized strengthgroup on the Post-test scores were lower than pre-test scores increasing treatment format, if nec 4. Administers the posttest Fig. Salkind (2018). Pg 185 The assumpt PREEXPERIMENTAL DESIGN: A ONE-GROUP PRE-TEST POSTTEST DESIGN PROBLEM Problem: Social desirability and acquiescence effects, arising from seeing the film, are possible rival explanations of the result. These testing effects (or sideeffects as they produce effects other than that intended) would affect the internal validity of the study. PRE-EXPERIMENTAL STUDY DESIGNS Pre-experimental designs have no random assignment of participants. No measure of, or control over extraneous variables. Questionable internal and external validity. 🙁 Cohen, D. S., (1996). Simpsons. Season 7 Episode 23 Kinds of Experimental Designs Betweensubjects design Withinsubjects design Mixed design Different participants in each group Same participants in each group Different participants in each group (e.g. Alternative Design 1) tested across occasions (e.g., Alternative Design 2) tested across occasions (e.g., Alternative Design 3) Between Subjects designs Different participants in each level of a factor e.g., naturally intact groups biological sex cultural background artificially derived groups age (young, old) identification with culture (low, high) OR Just random allocation! Alternative Design 1: Using a control group (A post-test only control group design) Film test test Method: Randomly assign university students into two groups, show the film to one group but not the other (control group) then test them again with a questionnaire on prejudice. Results: Prejudice scores were lower than control group. WHY IS THIS DESIGN STRONGER? A randomly assigned control group helps reduce problems relating to: Starts to increase internal and external validity by decreasing different types of biases and confounds Participant characteristics Bias in pool (Think WEIRD!) Selection bias in group allocation Time (external factors) Hopefully reducing measurement error and maximizing assessment of true score Example: 1-way Between Groups TV and aggression Male Female Ninja Turtles Behaviour Behaviour The independent variable is gender. (Nominal) The dependent variable is aggressive behaviour. (Ordinal) The hypothesis is that males behave more aggressively than females after watching a film with violence. Good if using intact groups (e.g., gender) ADVANTAGES OF BETWEEN SUBJECTS DESIGNS Need to look an irreversible changes (e.g., learning) Treatments (A, B) have carryover effects (e.g., A then B different from B then A) Using time-consuming treatments (e.g., therapy) More than one factor? Often more than one factor is required to explain some outcome (yes, life’s not simple) For example, reward depends on opportunity and performance Example: 2-way Between-Groups TV and aggression Male Behaviour Female Male Female Behaviour The independent variables are gender and film. (Nominal) The dependent variable is aggressive behaviour. (Ordinal) The hypothesis is that males behave more aggressively than females after watching a film with violence but not after watching a ‘non-violent’ film. Within Subjects designs Same participants in each level of a factor e.g., repeated measures therapy (pre,post) mood (+, -) Perceptual judgements related measures self esteem rating of child (father, mother) IQ (twins) Alternative Design 2: Testing change with a control group Pretest Pretest Film Post-test Post-test Within Subjects Pretest – Posttest Control Group Design Method: Randomly allocate a sample of university students into two groups, show the film to one group but not the other (control group) then test them again with a questionnaire on prejudice. Results: Change scores were lower in the control group. esign? First, How might be very usecause-andif you were earlier time e one-group llowing: ep 4 posttest is ministered n studying le strength. letion of the nt articipant’s True experimental designs control selection of subjects, assignment to groups, and assignment of treatments. For example, let’s look at one of the most popular of these designs, the pretest posttest control group design, which looks like this: Alternative Design 2: Problem? Step 1 Step 2 Step 3 Step 4 Random assignment of participants to a control group A pretest is administered No treatment is administered A posttest is administered Random assignment of participants to the experimental (or treatment) group(s) A pretest is administered A treatment is administered A posttest is administered For this design, the researcher would follow these steps: Pretest Film Post-test 1. Randomly assign the subjects to the experimental group or the control group Pretest 2. Pretest each group on the dependent variable Post-test 3. Apply the treatment to the experimental group (the control group does not receive the treatment) 4. Posttest both the experimental group and‘sensitize’ the control Problem: The pretest might have the students as to the ed strengthon the dependent variable (in another form or naturegroup of the study. Hence, social desirability and acquiescence format, if necessary) effect are still possible rival explanations of the result. The assumption here, and you are probably on to this, In other words, pretesting itself can produce testing effects en the preis that because the subjects are randomly assigned to eiwould affect the internalgroup, validity he primary that ther the control group or the experimental they areof the study. s no control equivalent at the beginning of the experiment. Any differ- ADVANTAGES OF WITHIN SUBJECTS DESIGNS large individual differences expected e.g., attitudes, psychological health studying change short duration experiments (e.g., perception) small sample size (same sample through all conditions) Mixed designs Same participants in each level of one factor, tested again in levels of another factor e.g., gender therapy (pre,post) limcher ams t ininthe uld is ed is ed is ed ple, here ferses, ver; ype e of that ent, s efment design is that sometimes it is not convenient or may even be impossible to administer a pretest. Under these conditions, you can use the posttest-only design. There are basically two disadvantages to using a posttest-only design. First, if the randomization procedures were not effective, the groups might not be equivalent at the start. Second, you cannot use the pretest to assign people to other experimental groups, such as high or low on some variable. These disadvantages may be of Pretest little consequence, yet they deserve some consideration. Alternative Design 3: 4 - group Solomon Design Film Post-test classical experiment The Solomon four-group design is extremely useful, but it is also expensive and time consuming. Pretest Post-test The last true experimental design is kind of the grand mommy and grand daddy of them all, the Solomon fourgroup design, as shown here: Step 1 Step 2 Step 3 Step 4 Random assignment of participants to a control group A pretest is administered Treatment is administered A posttest is administered Random assignment of participants to experimental or treatment group 1 A pretest is administered No treatment is administered A posttest is administered Random assignment of participants to experimental or treatment group 2 No pretest Treatment is administered A posttest is administered Random assignment of participants to experimental or treatment group 3 No pretest No treatment is administered A posttest is administered Film Post-test effect of pretesting Post-test Fig. Salkind (2018). Pg 186 ADVANTAGES OF MIXED DESIGNS Combines advantages of between groups design with those of within groups design Allows for investigation of interaction between a factor that is best handled by between groups design with that best handled by within groups design (e.g., attitude change between film vs no film control) HOW ‘GOOD’ IS THE CONTROL GROUP? ONE STRIVES TO HAVE A CONTROL GROUP THAT IS EQUIVALENT TO THE EXPERIMENTAL GROUP (CLONES WOULD BE IDEAL). THE BEST WAY IS TO RANDOMLY ASSIGN PARTICIPANTS INTO THE CONTROL AND EXPERIMENTAL GROUPS. IT IS SUPPOSED TO DISTRIBUTE NON-SYSTEMATIC VARIANCE, SUCH AS ‘INDIVIDUAL DIFFERENCES, RANDOMLY AND EQUALLY TO BOTH GROUPS. RANDOMISATION MIGHT NOT WORK WELL WITH A VERY SMALL SAMPLE (E.G., LESS THAN 20) AS, BY CHANCE, THE GROUPS MIGHT NOT BE EQUIVALENT. Quasi-experimental design EQUIVALENT CONTROL GROUP Sometimes it is not possible to randomly assign participants to a control group and an experimental group – as in the case of intact groups (e.g., head-injured patients versus control). Still, we can make groups ‘equivalent’ by selecting participants for a control group who matches those in the experimental group on ‘things that matter’ (e.g., age, sex, education and, in the above example, trauma of the experience). These ‘things’ depend on one’s knowledge of the area of research. the true experimental method has the most, and the quasiexperimental method is somewhere in the middle. The more control a design allows, the easier it is to attribute a cause-and-effect sequence of events. Another way in which these three designs differ from one another is the degree of randomness that enters into the design. You already know that the word random implies an of investigation equal and independentMethod chance of being selected, but that definition and concept can be applied beyond selection determines thetheinternal of a sample of subjects from the concept’s anda population externalto validity of importance in experimental design. quasi-experimental designs. Even though quasi-experimental designs will be discussed in Chapter 12, it is included here so you can see a comparison of all design types. Notice that many of these differences focus on the process of randomization of selection procedures, subjects, and assignment. SUMMARY Pre-Experimental Designs Use of a control group Pre-experimental designs are not characterized by randomisation ofa population, nor randomand selection of participants from participants do they include a control group.to Without either of these, the power of the research uncover the causal nature treatments aretohallmarks the study of the relationship between independent and dependent of a true experiment variables is greatly reduced, if not entirely eliminated. The point at which random assignment enters the process These designs allow little or no control over extraneous distinguishes different types of experimental designs from one another. variables that might be responsible for outcomes other than whatconstant), the researcher intended. For example, a parent Elimination (holding uses an old folk remedy (wearing garlic around the neck) Actually, different steps need to be taken to ensure the or minimisation, of other to ward off the evil spirits associated with a child’s cold. quality of true randomness in the best of all experimental possible explanations of the Lo and behold, it works! This is the weakest type of designs. effect. If thisexperimental is not possible, conclusion to reach because there is virtually The first step is one you know most about, the random these no comparison to show that the garlic worked better than selection of subjects from a population to formdistribute a sample. This (‘balancing’) is influences between groups. Table 11.1 Differences between pre-experimental, true experimental, and quasi-experimental designs. Condition Pre-Experimental Design True Experimental Design Quasi-Experimental Design Presence of a control group? In some cases, but usually not Always Often Random selection of subjects from a population? No Yes No Random assignment of subjects to groups? No Yes No Random assignment of treatment to groups? No Yes No Degree of control over extraneous variables? None Yes Some Fig. Salkind (2018). Pg 184 What's My Cue? Attention Orienting to Local/Global Stimuli Across the High and Low Subclinical Autism Spectrum Melanie J Murphy1, Madison Astbury1, Caszy Bartlett1, Rowena Bicknell1, Rebecca Chong1, Albert Delos Santos1, Maryam Farache1, Emma Foley1, Matthew Groenewegen1, Emily Grundy1, Salonika Hardie1, Ruth Honner1, Keri Justice1, Zoe Lazaridis1, Tahlia Liapis1, Alexandra Liddle1, Julian Maccioni1, Lauren Milano1, Natalie Mizzi1, Renee Morihovitis1, Robyn Pinto1, Connie Pizzi1, Neha Quai Hoi1, Doriano Raffaele1, Timothy Rodgers1, Madeleine Russell-Maynard1, Clementine Ward1, Lily Watson1, Emma Wood1, Sheila G Crewther1, Robin Laycock2. 1. Department of Psychology and Counselling, School of Psychology and Public Health, La Trobe University. 2. School of Health & Biomedical Sciences, RMIT University INTRODUCTION Given the evidence describing differences in attention processing in clinical Autism Spectrum Disorder (ASD), it is of particular interest to examine how the degree to which individuals exhibit ASD-like traits (ATs) can influence what aspect of the visual environment is attended to, and what features can engage and orient attention processing. Individuals with Autism Spectrum Disorder as well as subclinical populations with higher AT are reported to demonstrate a local bias in visual processing. Assessment of such processing typically requires explicit attention to global and/or local information. Aims: The current study explores attention to explicit global vs local information using Navon-letter stimuli, and then examines differences in involuntary/reflexive attention orienting by local/global compound arrow stimuli with different stimulus onset asynchrony (SOA) in a sub-clinical adult sample with high or low AT. We ask; Would a sub-clinical adult population with higher or lower autism-like traits show anomalies in explicit local/global processing using Navon letter stimuli? Does involuntary reflexive processing of local/global compound arrow stimuli differentially orient spatial attention? METHODS Participants: Seventy-nine participants recruited via social networks of undergraduate Psychology students at the La Trobe University Melbourne, Bendigo and Albury-Wodonga campuses were categorized as high or low AT based on Subthreshold Autism Trait Questionnaire (SATQ) scores. Inclusion criteria: No diagnosis of ASD, history of other neurological or developmental disorders and normal/corrected vision. Materials: Navon task: Required detection of the presence or absence of a target letter ‘A’ at that appeared at either local, global, or both levels. Compound Arrow-Cueing (Posner) task: Required the participants to respond to the appearance of a dot target presented to either side of a congruent and incongruent hierarchical arrow stimulus pointing left or right after a 250 or 750 ms SOA. Procedure: Participants completed the SATQ, and then the Navon and Posner Tasks in counterbalanced order. Target ‘present’ and ’absent’ response keys were counterbalanced for the Navon task. Participants were informed that the Posner arrow was irrelevant to target location. RESULTS Figure 1. Mean SATQ score for high and low AT groups. SATQ scores were significantly different for the High and Low AT groups. Posner Arrow Task: Mixed model ANOVA showed: Significant main effect for stimulus congruency (p <.01) and SOA (p <.01) Significant interaction effect between SATQ and congruency (p = 0.05) Interaction between SAQT and SOA was p =.056. Cueing effect present for Low AT group at 250ms SOA was not apparent with 750 ms SOA for congruent arrows, but still occurred in the High AT group. Greater global cueing effect for incongruent arrows present at 250 ms SOA for Low AT group, but to a lesser extent in High AT group. Trending (ns) toward a local cueing effect for incongruent arrows at 750 ms SOA for High AT group, but no cueing effect for Low AT group. DISCUSSION & CONCLUSIONS Would a sub-clinical adult population with higher or lower autism-like traits show anomalies in explicit local/global processing using Navon letter stimuli? A local bias was not observed for the High AT this Subclinical sample. The trend towards lower levels of global interference. Further investigation is required to examine magnitude of interference as symptomology changes along the AT spectrum. Figure 2a. Navon Task Stimuli Figure 2b. Local and Global Interference Effect for High and Low AT groups. Navon Task: In general both groups were more susceptible to global interference with detection of local stimulus. To a slightly less extent in the High AT group (not significant) Figure 3a. Posner Arrow Task Stimuli (above). Figure 3b. Arrow Cueing Effect effect High and Low AT groups. Does involuntary reflexive processing of local/global compound arrow stimuli differentially orient spatial attention? The Low AT group exhibited a shift from a global-cueing effect at 250ms to a more local-cueing effect at 750ms SOA. Non-predictive arrow cueing effects decrease with longer SOA in Low AT (Van der Hallen et al., 2014). For the High-AT group, the larger cueing effect for congruent arrows at 750ms SOA may indicate a slower capacity to override exogenous attention processes with top-down control. Results are consistent with an association between AT and a reduced utilisation of implicit visual orienting (Landry & Parker, 2013; Laycock et al., 2017). REFERENCES Landry, O., & Chouinard, P. A. (2016). J of Cognition and Development, 17(4), 584-595. Landry, O. and Parker, A. (2013). Frontiers in Human Neuroscience, 7, 833. Laycock, R., Chan, D., & Crewther, S. G. (2017). Frontiers in psychology, 8. Laycock, R., Crewther, S. G., & Crewther, D. P. (2007). Neuroscience and Behav Reviews, 31, 363 – 376. Mills, M., and Dodd, M. D. (2014). J of Experimental Psychology, 143 (4), 1429 – 1436. Navon, D. (1977 Cognitive Psychology, 9, 353 – 383. Plaisted, K., Swettenham, J., and Rees, L. (1999). J of Child Psychology and Psychiatry, 40 (5), 733 – 742. Van der Hallen, R., Evers, K., Brewaeys, K., Van den Noortgate, W., Wagemans, J. (2014). Psychological Bulletin, 22, 17-29. doi: 10.1037/bul0000004. Next Week Correlational Design