Research Methods: Applied Empirical Economics Lecture 4 PDF

Research Methods: Applied Empirical Economics Lecture 4: Regression Discontinuity Dr. M. van Lent | Leiden University September 12, 2024 1  Recap previous lecture Instrumental variables (IV) isolate random variation in trea...

Research Methods: Applied Empirical Economics Lecture 4: Regression Discontinuity Dr. M. van Lent | Leiden University September 12, 2024 1  Recap previous lecture Instrumental variables (IV) isolate random variation in treatment - Used in the context of experiments with non-compliance - But use can be generalized to all kinds of other applications You should be able to tell a story what exact process provides you with as-good-as-random variation in treatment - This requires knowledge of the treatment process Although there are formal tests, many arguments are judgmental: - Which event provides the instrument? - Is the variation due to the event really as-good-as-random? - Is the exclusion restriction satisfied? 2  Regression discontinuity (RD) With experimental data, selection bias is eliminated ex ante But with observational data, there are two potential routes to be taken to eliminate selection bias ex post: - Instrumental variables and regression discontinuity: Recognize events that are as-good-as-random - Difference-in-differences (and matching techniques): Make sure that you control for all variables that may be correlated with the treatment and the outcome variables → no omitted variables left… 3  Learning goals of this lecture Learn how to use regression discontinuity (RD) designs to obtain quasi experimental variation in a policy variable, while using observational data See chapter 4 in MM 4  Home protection and burglary- proof houses Do better protected houses reduce burglaries? Difficult to answer this question, omitted variables can be everywhere: - Houses with different protection levels are different in other dimensions - Bigger (and newer?) houses are better protected. - Location, expected benefits from breaking in, etc. -.. 5  Home protection and burglary- proof houses We need a treatment that leads to similar houses that randomly vary in how ‘burglary-proof’ they are. Policy: As of 2001, all newly build houses in NL need to have burglary-proof windows and doors. RQ: What is the effect of this treatment (better protected houses due to a change in policy) on the outcome (burglary rate)? Vollaard, B. and J. van Ours (2011), Does the regulation of built-in security reduce crime? Evidence from a natural experiment, The Economic Journal, 121(552):485-504 6 Around cutoff, treatment and control group very similar 7 Around the cutoff, it is like flipping a coin 8 Burglary rate 9 Burglary rate 10  Estimating treatment effect ρ Estimating the treatment effect is straightforward (with OLS): Yi = α + β cyi + ρ Di + εi 11  Examples where RD can be used Class size thresholds Date of birth and school starting date Test scores and education track admission Income thresholds for receiving government funds 12  RD: key idea There’s a regression with a discontinuity in it Beyond a cutoff, the treatment is turned on or turned off Subjects around the cutoff point are on average similar in their characteristics - You can test for this! (‘balancing test’) Only thing that changes is whether the subject received the treatment 13  Omitted variable bias gone? As long as covariates determining treatment and outcome change smoothly, they do not affect the estimated impact One way to test for this is by closely looking at any other policy parameters that may lead to cutoffs - Think e.g. about possible other regulations that started in 2001 and may have affected burglary! But one may also address this question with formal tests: - Any other covariate should not change discontinuously! - A test on the treatment effect would be to add covariates (X) and see what happens to ρ Yi = α + β cyi + ρ Di + γ Xi + εi 14  RD as a Local Average Treatment Effect (LATE) RD identifies treatment effects on a limited support of the ‘running variable’ → around the cutoff! This means that the RD estimate is a “LATE” - Recall that instrumental variable estimation also provides LATE estimates: the treatment for those that were affected by the instrument! Treatment effects may vary over the support of covariate determining eligibility - Again, this is largely a matter of coming up with good arguments! - Some studies do provide RD estimates at various places of the support. - Class size effects: Leuven, Oosterbeek and Ronning, 2008, Quasi-experimental Estimates of the Effect of Class Size on Achievement in Norway, Scandinavian Journal of Economics, 110(4), 663–693 15 Multiple cutoffs affect class size (and in turn educational achievement) 16 The importance of functional form of the running variable If the relationship with the outcome and the running variable is not flat, treatment and control groups are not on average similar! It is important to have feeling for the data – and not just stick to a flat line with a discrete jump on top of it ! To start with, it is instructive to plot averages of the outcome variable for intervals of the running variable You next choose a flat line, a linear one or a ‘polynomial’ and see how it fits the data. 17 Simplest case: flat line 18 A linear setup 19 Implications for specification The Linear case Yi = α + β Ri + ρ Di + γ Xi + εi Polynomial case Yi = α + β1 Ri + β2 Ri2 + ρ Di + γ Xi + εi No clear discontinuity at all (if specified as polynomial) 20  Choosing functional form and the bandwidth One way to choose the ‘optimal’ functional form is by increasing the number of polynomials of the running variable - From linear, to quadratic, third power etc.. - At some point, the fit to the data should be fine (check with the R-squared and t-values) The other is by changing the bandwidth around the cutoff. - As RD is a LATE, it makes sense to zoom in to treatment and controls that are alike - We then return to the model with a flat or linear function for the running variable But: narrowing the window comes at a cost: less data! - Tradeoff: more data points increases precision, more data points makes C and T less comparable. - Articles therefore usually show outcomes for multiple bandwidths! 21  The key assumption: no manipulation RD assumption: subjects cannot manipulate the running variable that determines the treatment assignment. - This may be the case when subjects have an interest to do so! - Think of speeding up construction project to avoid regulation You should come up with a story whether there was manipulation or not - Think of information set: was policy known, was there room to adapt? You can also run falsification tests: look at variables that should be unaffected by the treatment without manipulation - Example: Compare the number of houses (and type/size etc.) constructed the year (or last months) before and after the new policy 22  Summary (sharp RD) Identify exogenous variation induced by a policy/treatment. The precise cut-off for policy/treatment eligibility is considered random. Key assumption: Observations close to the cut-off are the same except from some being treated and others not. Check covariates (balance test) Check number of observations What to choose: Functional form Bandwidth Typical check: how robust is the treatment effect to different (arbitrary) choices of functional form and bandwidths. 23  Sharp or fuzzy RD? So far, we have looked at treatments that constitute a switch on/off at cutoff → “sharp RD” But in many cases, there is a discrete change in treatment intensity that changes the probability of receiving the treatment - Some minimum entrance exam score to be eligible to Boston exam school (MM, pp. 166) - A maximum income to apply for a grant/subsidy In this context, there is a “fuzzy RD” Methods that are needed to infer treatment effects with fuzzy RD are equivalent to Instrumental variable estimation! - Recall from IV with field experiments: treatment dilution and treatment migration! 24  Fuzzy RD as IV (recall from previous lecture!) To test whether there is an effect at the cutoff anyway, you can start by estimating a reduced form model like this: Yi = α0 + β Ri + ρ Di + γ0 Ai + e0i 25  Fuzzy RD as IV (recall from previous lecture!) To test whether there is an effect at the cutoff anyway, you can start by estimating a model like this: Yi = α0 + β Ri + ρ Di + γ0 Ai + e0i - In this case, we have a linear function of the running variable - A = other covariates ρ can be interpreted as the Intention-to-treat (ITT) effect of the cutoff - It may be driven by large or small changes in the treatment around the cutoff 26  Fuzzy RD as IV If there is an ITT effect, you can try TSLS. You then first estimate the impact of the cutoff on the probability of the treatment, T, while including A as controls (first-stage regression): Ti = α1 + β1 Ri + Ф Di + γ1 Ai + e1i Next, the first-stage fitted values of T can be used in the second-stage equation Yi = α2 + β2 Ri + λ2SLS T෡i + γ2 Ai + e2i 𝜌 Thus, λ2SLS = Ф 27 Example of fuzzy RD: effect of having to take a resit on academic performance Suppose we have a model like AcPerfi = α + β Gradei + ρ Resiti + εi with AcPerf as later academic performance of student i, Grade for the initial grade in the current course and Resit for having a resit (=treatment!) Gradei is the initial grade and in the RD the “running variable” Resit not only determined by initial grade, but also by later discussions with the lecturer → Fuzzy RD, so TSLS needed! 28  Example, continued First stage: Resiti = α1 + β1 Gradei + Ф Di + ε1i This yields the predicted value of Resit Second stage: ෣ i + ε2i AcPerfi = α2 + β2 Gradei + λ 𝑅𝑒𝑠𝑖𝑡 Note: You can also add covariates here, more polynomials of Grade or vary the bandwidth! 29  Summarizing RD is another way of eliminating selection bias using observational data RD produces clear graphical evidence of treatment effects RD can be combined with IV when discrete changes around the cutoff involve changes in treatment probabilities To obtain credible RD estimates, it takes: - A clear, appealing policy rule (that is not interfered by other ones) - No manipulation of the running variable - A suitable specification and bandwidth 30  Example: Fuzzy RD Ganguli (2017): Saving Soviet science: The impact of grants when government R&D funding disappears RQ: How do research grants affect science. Outcome: future publications Treatment: receiving a grant The eligibility criteria are clear: grant 1: 3 or more papers published, grant 2: impact factors. However people have to apply for the grant in order to receive it. Instrument: Eligible for the grant. 31  Example: Fuzzy RD Ganguli (2017): Saving Soviet science: The impact of grants when government R&D funding disappears Just a raw comparison between those receiving the grant and not receiving the grant would be misleading. - Better scientist are more likely to be eligible for the grant and are therefore more likely to obtain the grant. In addition: better scientist have better science outcomes in the future. Therefore simple OLS would likely overestimate the treatment effect. - Fuzzy RD: First stage: being eligible regressed on receiving the grant. Second stage: estimate of ‘receiving the grant’ in first stage regressed on outcome. 32 Next time: Difference- in-differences! 33  How does this all work in STATA? You can start by making scatterplots of value averages of treatment incidence and outcome variables, with intervals of the running variable on the X-axis - In STATA: “scatter” or “graph toway scatter” - This shows you whether there are cutoff effects and what specification you may need In case of sharp RD: just estimate OLS functions (“regress” or “reg”), using an adequate number of polynomials In case of fuzzy RD: use IV (“IVregress” or “IVreg”), using an adequate number of polynomials Robustness can be tested by - changing the bandwidth : regress y r d if (r > r_min) & (r < r_max) - Increasing polynomials: regress y r r_squared d - Adding covariates: regress y r d x 34 Example: the effect of winning previous election on current election 35  Explanation variables of Lee et al. Margin: margin of victory, the “running variable” Treat: dummy if margin > 0 Share = the winning share in the next election Tmargin = treat * margin (to allow for discontinuity around threshold) 36 Estimation results 37 Estimation results Setting the bandwidth 38 Estimation results Treatment effect 39 Estimation results Running variable (linear) 40 Estimation results Running variable, asymmetric effect 41