Pre-Post Analysis: Empirical Methods

Empirical Methods

• Observational studies can be misleading due to correlation vs. causation.
• Causal estimates are important for determining policy.
• A few methods used are Randomized Control Trials (RCT), Non-Experimental, Quasi-experimental (or Natural Experiments), Difference-in-Differences, and Regression Discontinuity Design.
• Internal validity reflects how well a study isolates the causal effect of a treatment by minimizing biases and confounding factors.

Pre-Post Analysis

• Pre-Post Analysis is comparing the world before and after an intervention.
• A school introduces a free after-school tutoring program for students as an intervention.
• Student math test scores are the metric measured for both pre- and post- intervention.
• Potential problems with this analysis are other factors affecting test scores, regression to the mean, selection bias, and the observation effect.
• It is not possible to compare the blue line and the green line in the post-intervention in a Pre-Post Analysis.
• The counterfactual world is not visible.
• There's no visibility into the world where students didn’t get tutoring.
• The counterfactual is an alternative scenario which didn't occur but represents what would have happened without intervention.

Counterfactual

• Finding individuals/areas that are the same or similar to the individuals/areas of interest helps deduce a counterfactual.
• Changes, growth, or trends are compared and contrasted between treated individuals against the growth of untreated individuals.
• The best way to get counterfactual involves randomized controlled trials: research picks who gets treated and who doesn't get treated.
• Because of randomization, the Difference-in-difference method is used.
• Regression discontinuity is another method, though these are for later.
• As an example, find a different school that has similar quality students but no tutoring program.

Parallel Trend Assumption

• Absent treatment, the change in outcome for treatment and control groups would have been the same over time.
• Absent treatment, the difference between treatment and control groups would have remained the same in the post-period as in the pre-period.
• As an example, the school that implemented the tutoring program didn’t implement the program, they would show a similar trend with the comparison school.
• If the Parallel Trend Assumption is violated, the estimated treatment effect in a Difference-in-Differences analysis may be biased.
• The control group no longer provides a valid counterfactual for the treated group.

Difference-in-Difference

• Differences-in-differences studies the effects of policy changes which impact only some of the individuals/areas.
• Examples of this include women’s labor force participation rates in states that required paid parental leave versus those that did not, before and after the policy went into effect, and college enrollment increase in Georgia after 1993 merit aid program compared to neighboring Florida (which did not implement a similar policy).

Difference-in-Difference Formula

• A formula can estimate the causal impact of a policy: DifferenceTreated – DifferenceControl = (PostTreated - PreTreated) – (PostControl - PreControl).
• Covered are simple differences-in-differences cases such as two states or two groups, with no heterogeneous treatment effects.

Practice Example: The Affordable Care Act (ACA) and Medicaid

• The Affordable Care Act (ACA) was implemented in 2014.
• Some states expanded Medicaid, while others didn't.
• There is interest in assessing the effect that Medicaid expansion had on employment rates among low-income workers.
• Data was collected on the employment rate (percentage of low-income adults employed) before and after the ACA's Medicaid expansion in two groups.
• Treatment Group: States that expanded Medicaid.
• Control Group: States that didn't expand Medicaid.
• The following numbers are pre-expansion on the left, and post expansion on the right: Control States (No Medicaid expansion): 60 / 59, Treated States (Medicaid expansion): 62 / 57.
• Calculate the Difference-in-Differences (DiD) estimate of the effect of Medicaid expansion on employment.
• After calculating, one must interpret the result: Does Medicaid expansion seem to have affected employment, and if so, in what direction?
• Assess what potential threats there are to the validity of the DiD estimate.

Fracking and Education

• The research questions are how much the potential externality of fracking is on students and assessing the short-term and long-term effect of fracking.
• The hypothesis is that the exposure to fracking activities near schools negatively impacts students' academic performance in both the short and long-term.
• The outcomes include attendance and state-level test results.
• Fracking extracts natural gas and oil from deep rock formations named shale.
• Fracking has diverse potential environmental harm.
• Those who could be affected by contamination are students who study near well sites.
• Fracking development increased in the early 2000s in Texas.
• Some wells are closer than other wells to certain schools, and some wells are not close to any schools.
• The assumption is that Texas schools are similar except for the distance to the wells => Control group: schools that don’t have a well nearby, Treatment group: schools that have a well nearby.

Announcements

• Reading Memo 2 is uploaded.
• "Effects of the Affordable Care Act on Part-Time Employment" by Dillender, Heinrich, and Houseman (2022).
• The same format is used, being more than 1 page, less than 2 pages, double spaced, with paragraph summaries and comments in bullet points while having a minimum of 3 comments.
• Equilibrium will be revisited next class.