CS2008 2024 Fundamentals of Research - Experimentation PDF

CS2008 Fundamentals of Research Week 10.2: Experimentation 1  What You Should Learn Today A little bit about the history of experimentation Basics of experimental design – Manipulation – Control...

CS2008 Fundamentals of Research Week 10.2: Experimentation 1  What You Should Learn Today A little bit about the history of experimentation Basics of experimental design – Manipulation – Control 2 Brief History of Experimentation Experiments used in 11th century Arabia More formally, the experimental method developed from the scientific advances of the 16th and 17th centuries (e.g., Galileo Galilei) Allowed Sir Isaac Newton (1643 - 1727) to proclaim “the qualities of bodies are only known to us by experiments” Sir James Lind (1716-1799) Scottish physician and the pioneer of naval hygiene Conducted one of the first documented clinical trials (experiment) in 1747 Sought a cure for scurvy, but he didn’t know what would cure it Conditions– Each received the same typical naval diet (bread, butter, beans, & beer) Group 1: Quart of cider Group 2: 25 drops of vitriol (sulfuric acid) Group 3: 6 spoonfuls of vinegar Group 4: ¼ litre of seawater Group 5: 2 oranges and 1 lemon Group 6: Spicy pepper paste and ½ litre of barley water  Scurvy Experiment Condition 5 stopped after 6 days of treatment when they ran out of fruit, but by then… “…the most sudden and visible good effects were perceived from the use of the oranges and lemons; one of those who had taken them, being at the end of six days fit for duty.” A treatise of the scurvy, 1753 Experiment – Contemporary Example Which textbook improves student learning the greatest?  Simple Design Dependent variable: Degree of student learning – Can be operationalized in different ways Independent variable: Type of textbook given to students – Condition A: Textbook A – Condition B: Textbook B – Condition C: Textbook C  Ruling Out Alternatives The influence of the IV, with all other things being equal Ceteris paribus Equivalence by holding things constant – Same students (age, education level, aptitude, gender, socioeconomic status, course load, etc. )? – Same professor? – Same university? – Same test? – Same time/date of class sessions?  “Classic” vs. “Modern” Experiments 19th and 20th century classical experimentation – Practice of holding everything constant except the one variable under consideration Move to the modern form – Sir Ronald A. Fisher’s The Design of Experiments (1935) “Experimental observations are only experience carefully planned in advance” Not controlling all potential variables, but rather manipulating levels or amounts of selected independent variables in order to examine their influence on dependent variables Equivalence through randomization Why Do We Need Experimental Designs? Variability in the world – We wouldn’t need experimental design If all units (e.g., students, teachers, and schools) were identical, and If all units responded identically to treatments Experimental designs control variability so that the effects of our independent variable can be isolated and observed on the dependent variable Advantages of Experimental Methods Provide evidence of causality – Non-experimental methods have greater difficulty in establishing causation – Provide a high level of internal validity (if done well) Greater control – Over the environment – Over variables – Over participants Relatively lower cost Support replication Potential Disadvantages of Experiments Artificiality – Ecological validity Researcher bias (e.g., expectation bias) – Can be overcome with training, protocols – Double-blind design Limited scope  The Basics Key concepts for experimental research – Manipulation and control Manipulation of the independent variable (IV) is key to experimentation Manipulate IV and then observe changes in the DV for conditions and compare results Condition Dependent Independent 1 Variable Variable Compare Results Condition Dependent 2 Variable  Control In the scientific method, control allows for meaningful and valid comparisons to be made Concepts are examined at the operational level as varying dimensions/levels of IVs Controlling variables and attributes in the experimental setting so that adequate comparative conclusions can be reached Ceteris paribus – How does that work?  Control groups Experimental designs separate research participants into treatment groups (i.e., conditions) Experimental groups = treatment groups = conditions – Each receives some form of controlled stimulus Control group – Group of participants to whom no experimental stimulus is administered, but who should resemble the experimental group in all other respects – Comparison of the control group to the experimental group(s) reveals the effect of the stimulus Control of Confounding Variables Confounding variables create alternative explanations – Also called extraneous variables Treatment group comparability is key Three methods of controlling confounds associated with participants 1. Matching 2. Randomization 3. Inclusion of confounding variable Control of Confounding Variables 1. Control by matching Some sources of variation may be eliminated by matching – Identifiable individual differences – District or school effects However – Matching is only possible on observable characteristics – Perfect matching is never possible – Matching limits generalizability by reducing natural variation Control of Confounding Variables 2. Control by randomization Assignment of experimental subjects to experimental conditions and control groups at random Randomization makes groups equivalent (on average) on all variables (known and unknown, observable or not) Randomization controls for the effects of all (observable or non-observable, known or unknown) characteristics Randomization is a “precaution against disturbances that may or may not occur and that may or may not be serious if they do occur” (Cochran and Cox, 1957, p. 8) Degree of similarity can be checked Control of Confounding Variables 3. Control by inclusion of confounding variable Increases sensitivity of experimental design Allows for examination of interactions However – Increases complexity – Can increase number of participants needed – Can increase costs Q: Can you articulate the difference between random selection and random assignment? Q: When is each more likely to be used?  Upcoming More about Experimentation next week Tutorials on Friday (Hypothesis testing) – Please do the pre-reading 22