Causal Analysis Randomized Experiments PDF

Causal Analysis Randomized Experiments Michael Gerfin Blaise Melly University of Bern Spring 2024 Contents of this chapter 1. Introduction 2. Randomization 3. Examples for experiments 2 / 29 Introduction Refresher What is the identification problem? E[Yi |Di = 1] − E[Yi |Di = 0] = E[Y1i |Di = 1] − E...

Causal Analysis Randomized Experiments Michael Gerfin Blaise Melly University of Bern Spring 2024 Contents of this chapter 1. Introduction 2. Randomization 3. Examples for experiments 2 / 29 Introduction Refresher What is the identification problem? E[Yi |Di = 1] − E[Yi |Di = 0] = E[Y1i |Di = 1] − E[Y0i |Di = 1] {z } | {z } | observed difference AT T + E[Y0i |Di = 1] − E[Y0i |Di = 0] | {z } selection bias The selection bias arises if treatment and control group differ in Y0 Claim: Random assignment makes the selection bias zero 3 / 29 Randomization 1. Introduction 2. Randomization 3. Examples for experiments 4 / 29 Randomization What does random assignment do? Consider the following causal question: what is the impact of a deductible in health insurance on health care expenditures (hce) There is an obvious confounder: health status influences deductible choice and hce This is illustrated in the following DAG health status deductible hce 5 / 29 Randomization What does random assignment do? Assume we can run a randomized experiment Then the deductible level is determined by a lottery This changes the DAG as follows health status lottery deductible hce 6 / 29 Randomization What does random assignment do? Random assignment makes the treatment independent of all relevant factors in the causal model Hence, random assignment makes Di independent of potential outcomes, denoted as {Y1i , Y0i } ⊥ ⊥ Di This independence in turn implies that E[Y0i |Di = 0] = E[Y0i |Di = 1] E[Y1i |Di = 0] = E[Y1i |Di = 1] Then AT T equals the simple mean comparison AT T = E[Y1i |Di = 1] − E[Y0i |Di = 1] = E[Y1i |Di = 1] − E[Y0i |Di = 0] = E[Yi |Di = 1] − E[Yi |Di = 0] 7 / 29 Randomization Random assignment solves the selection problem This goes even further E[Y0i |Di = 0] = E[Y0i |Di = 1] = E[Y0i ] E[Y1i |Di = 0] = E[Y1i |Di = 1] = E[Y1i ] Therefore E[Y1i − Y0i |Di = 1] = E[Y1i − Y0i ] = AT E AT T is the same as the average effect in whole population, denoted by AT E Randomization makes treatment and no-treatment group comparable in all pre-treatment characteristics, including Y0 8 / 29 Randomization Regression analysis of experiments Start from the expression for the observed outcome Yi = Y0i + (Y1i − Y0i ) · Di Now suppose that Y1i − Y0i = τ is constant. Then Yi = |{z} α + |{z} τ · Di + E[Y0i ] Y1i −Y0i ηi |{z} Y0i −E[Y0i ] This looks very much like a regression model, and the treatment effect is given by τ (under the constant-effect assumption) 9 / 29 Randomization Regression analysis of experiments Evaluating the conditional expectations with the treatment status switched on and off yields E[Yi |Di = 1] = α + τ + E[ηi |Di = 1] E[Yi |Di = 0] = α + E[ηi |Di = 0] E[Yi |Di = 1] − E[Yi |Di = 0] = τ + E[ηi |Di = 1] − E[ηi |Di = 0] {z } | selection bias E[ηi |Di = 1] − E[ηi |Di = 0] = E[Y0i |Di = 1] − E[Y0i |Di = 0] Selection bias amounts to correlation between Di and ηi This correlation reflects the difference between Y0 in the treatment and in the no-treatment group 10 / 29 Randomization Regression analysis of experiments One of the assumptions in the classical linear regression model states that the error is (mean) independent of the regressors Here the single regressor is the treatment indicator Di , the error term is ηi Under mean independence, OLS is unbiased for τ This is what randomization is about: it ensures that mean independence holds, i.e., randomization implies that E[ηi |Di = 1] − E[ηi |Di = 0] = 0 There is no need to control for other covariates X (Why?) 11 / 29 Examples for experiments 1. Introduction 2. Randomization 3. Examples for experiments 12 / 29 Examples for experiments Examples of experiments Zooming to Class, Kofoed et (2024), Economic Insights The impact of computer usage on academic performance, Carter et al (2017), Economics of Education Review RAND Health Insurance Experiment Project STAR (Student-Teacher Achievement Ratio) in Tennessee Two randomized trials on the effect of providing job training on later labor market outcomes: NSW (National Supported Work experiment) and JTPA (Job Training Partnership Act) (will analyze them later) 13 / 29 Examples for experiments Zooming to Class The United States Military Academy at West Point, New York is a highly selective, public liberal arts college West Point students have little control over their academic schedules and peer networks West Point also randomly assigns students to academic instructors Curriculum is uniform and graded events are identical in content and delivery Ideal situation to set up a randomized trial for evaluating online teaching Kofoed et al (2024), Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19, Economic Insights 14 / 29 Examples for experiments Zooming to Class: Design Course in question is manadatory (Principles in Economics), hence no selection Randomly assigned 551 students across twelve instructors Each instructor agreed to teach half of their four section teaching load online and half in-person therefore, possible to control for instructor effect, because each taught online and in-person Highly disciplined students, therefore no compliance issues Kofoed et al (2024), Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19, Economic Insights 15 / 29 Examples for experiments Credibility of experimental design First question to ask about randomized experiment is whether randomization successfully balanced the subjects characteristics across treatment groups Common: compare pre-treatment outcomes (if observed) and other covariates across treatment groups Test for statistically significant differences 16 / 29 Examples for experiments Zooming to Class: Results I Kofoed et al (2024), Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19, Economic Insights 17 / 29 Examples for experiments Zooming to Class: Results II Kofoed et al (2024), Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19, Economic Insights 18 / 29 Examples for experiments Zooming to Class: Results III Kofoed et al (2024), Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19, Economic Insights 19 / 29 Examples for experiments Use of Electronic Devices on Learning Outcomes Use of electronic devices in class may have positive and negative effects on students’ learning outcomes Positive: ability to take notes faster, store copies of notes, and carry the notes with you at all times Negative: potential distractions from web-surfing, e-mail checking, and electronic chatting with friends Conduct experiment that randomly allowed students to access their laptop and tablet computers during an introductory economics course at the United States Military Academy Classrooms divided into a control group or one of two treatment groups. 1. treatment group: students may use laptops and tablets without restriction 2. treatment group (“modified tablet”): only permitted to use tablets, but the tablet had to remain flat on the desk surface Carter et al (2017), The impact of computer usage on academic performance: Evidence from a randomized trial at the United States Military Academy, Economics of Education Review 20 / 29 Examples for experiments Experimental Design Carter et al (2017), The impact of computer usage on academic performance: Evidence from a randomized trial at the United States Military Academy, Economics of Education Review 21 / 29 Examples for experiments Credibility of Experimental Design Carter et al (2017), The impact of computer usage on academic performance: Evidence from a randomized trial at the United States Military Academy, Economics of Education Review 22 / 29 Examples for experiments Effect of Use of Electronic Devices on Learning Outcomes Carter et al (2017), The impact of computer usage on academic performance: Evidence from a randomized trial at the United States Military Academy, Economics of Education Review 23 / 29 Examples for experiments RAND Health Insurance Experiment (HIE) Randomly assigned 2,000 families from six US cities to different insurance coverage plans Ran from 1974-1981 Co-payments are 1 2 3 4 5 6 free care 25% co-payment 50% co-payment 75% co-payment 95% co-payment (catastrophic plan ) 95% co-payment for outpatient care, free inpatient care Different levels of stop loss either 5%, 10% or 15% of household income with maximum of $1000 24 / 29 Examples for experiments HIE: credibility Source: Mastering Metrics 25 / 29 Examples for experiments HIE: results Source: Mastering Metrics 26 / 29 Examples for experiments Project STAR Tennessee STAR: Pioneering randomized study designed to estimate the effects of smaller classes in primary school STAR experiment was unusually ambitious and influential Cost about $12 million Implemented for a cohort of kindergartners in 1985/86 Duration four years, i.e., until the original cohort was in third grade (about 11,600 children) The experiment assigned students to one of three treatments: small classes with 13-17 children, regular classes with 22-25 children and part-time teacher’s aide regular classes with full time teacher’s aide Schools with at least three classes in each grade could choose to participate in the experiment 27 / 29 Examples for experiments Credibility of experimental design 28 / 29 Examples for experiments Impact of class size on test scores 29 / 29