Millennium Villages Project Evaluation

• Observational alternatives to experiments include selection on observables, selection on unobservables, and difference-in-differences (DID).
• Selection on observables means the treatment and control groups differ from each other only with respect to observable characteristics.
• Selection on unobservables means the treatment and control groups differ from each other in unobservable characteristics.
• Exogenous variables inducing variation in treatment can be analyzed using instrumental variables (IV).
• A known selection mechanism can be analyzed using regression discontinuity designs (RDD).
• Observing treatment and controls before and after treatment is the basis of difference-in-differences (DID).

Difference-in-Difference Design (DID)

• Aims to estimate causal effects by using differences between groups.
• Requires finding treatment and control groups that are similar in every way except for receiving the treatment.
• Without randomization, identifying such groups is difficult.
• Relies on the assumption that in the absence of treatment, the difference between treatment and control groups is constant over time, also known as parallel or common trends.
• This assumption relaxes the stringent requirement that treatment and control groups be almost identical.
• Uses observations in treatment and control groups, both before and after the treatment, to estimate a causal effect.
• Pre-treatment difference between the groups is the 'normal' difference.
• Post-treatment difference is the 'normal' difference plus the causal effect of treatment.
• The difference-in-differences is the estimated causal effect.
• Heavily relies on common or parallel time trends, so visual inspection of the data is essential.
• The basic design involves two groups and two time periods.
• The canonical version involves two time periods and two groups, treatment timing occurs at the same time for all those treated.
• More current DID applications use data from more than two time periods and have treatments occurring at different times.
• Parallel trends is the key assumption for any DID strategy is.
• The outcome in the treatment and control groups must follow the same time trend in the absence of treatment.

Millennium Villages Project

• A joint project by the United Nations Development Program (UNDP), the Earth Institute at Columbia University, and an NGO called "Millennium Promise”.
• A large, expensive intervention at 15 sites in rural sub-Saharan Africa.
• It seeks to show that "people in the poorest regions of rural Africa can lift themselves out of extreme poverty in five years' time" (MVP 2007).
• The intervention was launched first in Sauri, Kenya in 2004, and later at a number of sites across sub-Saharan Africa.
• Interventions included distribution of fertilizer, school construction, insecticide-treated bednets, HIV testing, microfinance, electric lines, road construction, and water and irrigation.
• Designed in line with the "big push" theory of economic development.
• A before and after comparison showed that the share of mobile phone ownership increased substantially in the Millennium Village between 2005 and 2008.

Measuring Program Impact

• It's necessary to ask what happened at sites that received a project's package intervention relative to what would have happened in the absence of the project.
• "Absence of the Project" means what would have happened without that specific project.
• A control group is needed that is comparable to the treated group in order to assess the absence of the program.
• With observational data, it can be difficult to construct a control group “ex-post” (after the intervention).
• The term "comparable” depends on the assumption of the specific estimation method used (DID, IV, RDD).

Adding Possible Control Groups

• Changes at the sites need to be compared to broader trends in the countries where the Millennium Villages are located.
• In Kenya, treated villages follow the same trend as the rest of the country.
• The rest of Kenya may not had the MVP.
• The mobile ownership would have risen at the MVP sites with or without the project.
• The counterfactual, or what would have happened without treatment, would likely have been a similar increase in mobile ownership as what was found in similar villages in the rest of Kenya.

Difference-In-Differences Graphically

• Measures the outcome of interest (y) in two time periods.
• Dots represent means of the outcome for each group in each time period.
• lines connecting the dots are just for visualization purposes.
• Treatment occurs, but only one group is treated.
• Compares mean difference before and after treatment.
• The control group captures any common changes in the treatment and control groups, which shows the counterfactual trend in the treatment group.

Key Assumption of DID

• Parallel trends are the main assumption for any DID strategy.
• This means the outcome in the treatment and control groups would follow the same time trend in the absence of treatment.
• It does not mean that they must have the same mean (or level) of the outcome variable.

Testing Assumptions of DID

• For parallel trends, is usually obtained by showing that in the period before treatment, the two groups developed in a similar manner.
• Better evidence is gathered if we have observations from several points in time.
• For common shocks: You must show that other policies or changes coincided with the treatment period, and these affected the treatment and control groups in the same way.
• You must convince the reader that nothing else happens at the same time as the treatment that would affect the control and treatment groups differently.

Example: New Jersey Minimum Wage Increase

• Treatment (T): Higher minimum wage.
• Outcome (Y): Employment.
• April 1, 1992, NJ increased the state minimum wage from $4.25 to $5.05.
• Card & Krueger (1994) wanted to measure how this change affected employment.
• Possible evaluation strategy: compare employment in NJ in 1994 with employment in March 1992 (before/after).
• Economy wide changes between 1992-94 could be a confounder.
• Pennsylvania (PA’s) minimum wage stayed at $4.25, and was used as a control.
• Card & Krueger (1994) surveyed about 400 fast food stores both in NJ and in PA before (February) and after (November) the minimum wage increase.
• Macroeconomic trends captured by using the control group in PA.

Card & Krueger (1994) DID Equations

• Yist: employment at restaurant i, state s, time t.
• E[yist|s = NJ, t = Feb]: mean employment in February.
• E[yist|s = NJ, t = Nov]: mean employment in November.
• E[yist|s = NJ, t = Nov] – E[yist|s = NJ, t = Feb] = Difference 1: difference in employment in NJ, the treated area.
• E[yist|s = PA, t = Feb] = mean employment in February.
• E[yist|s = PA, t = Nov] = mean employment in November.
• E[yist|s = PA, t = Nov] – E[yist|s = PA, t = Feb] = Difference 2: difference in employment in PA, the control area.
• The population DID is the treatment effect.
• The sample analog is the DID estimator.
• They found, employment increased in New Jersey, against what would have been expected.

Using Regression for the DID

• In the 2x2 case the regression model is Yit = α + βtreatedi + γaftert + δtreatedi ∗ aftert + uit.
• Treated = 1 if the observation is in the treatment group, 0 otherwise.
• After = 1 if the observation is from the after period, 0 otherwise.
• Treated*after = 1 if the observation is in the treatment group AND observed after the treatment.
• Treated and After are dummy variables; their product is called an interaction term.
• Alpha is referred to as the intercept or constant term. Given the New Jersey and Pennsylvania example
• Yist = α + βNJs + γNovt + δNJs ∗ Novt + uist.
• NJ = 1 if the observation is in New Jersey, the treatment group, 0 otherwise (regardless of the time period).
• Nov = 1 if the observation is from the after period, 0 otherwise (regardless of the state).
• NJ*Nov = 1 if the observation is in New Jersey observed after the treatment.

With Regression

• NJ before: E[yist | NJ = 1, Nov = 0] = α + β
• NJ after: E[yist | NJ = 1, Nov = 1] = α + β + γ +δ
• PA before: E[yist | NJ = 0, Nov = 0] = α
• PA after: E[yist | NJ = 0, Nov = 1] = α + γ
• Assuming that E[uist | NJ, Nov] = 0.
• DID = (NJ after – NJ before) – (PA after – PA before).
• NJ after – NJ before = (α + β + γ +δ ) – (α + β) = γ +δ.
• PA after – PA before = (α + γ) – α = γ.
• DID = (NJ after – NJ before) – (PA after – PA before) = δ.
• Estimating the regression model using OLS produces the DID estimate and standard errors which is very convenient.

DID Assumptions

• The main assumption is that the outcome in the treatment and control groups would have followed the same time trend in the absence of treatment (parallel trends.)
• To support the key DID assumption that a counterfactual situation cannot be directly observed, one has to show two things:.
• Parallel pre-trends: Show that in the period before the treatment, the two groups developed in a similar manner, i.e., followed a parallel trend. This is more convincing with several observations in time.
• Common shocks: Show that other policies or changes coinciding with the treatment period affected the treatment and control groups in the same way.
• Alternatively, convince the reader that nothing else (major) happened at the same time as the treatment that would affect the control and treatment groups differently.

Support for Parallel Trends

• Is shown by looking at pre-treatment trends.
• Even if pre-trends are the same one still must worry about other policies or changes coinciding with the treatment.
• Policies: Were there any other unemployment policies implemented in PA (the control group) during the studied period?
• It is very important for the researcher to be familiar with the institutional details of the reform/policy change:
• What macroeconomic vents that took place during the studied period, might affect T and C differently?
• e.g., consider a local recession in PA not affecting NJ.
• There must be no spillover effects of treatment.
• Group compositions also cannot change because of the treatment (if using repeated cross-sections).
• Yet another technical assumption: group composition must not change because of treatment
• That is, people do not move disproportionally from the unaffected to the affected group, or vice versa, because of the treatment).

Staggered or Differential Treatment Timing

• In this case, most DID applications exploit variation across groups of units that receive treatments at different times.
• In some applications, all units are eventually treated, while in others, there is a control group that never gets the treatment.

Curry and Walker Case Study (2011)

• Looked at traffic congestion and infant health.
• Examined the effects on health of infants relating to reduced traffic congestion resulting from E-ZPass.
• The research question asked: How does a reduction in air pollution affect the health of infants? The "bigger” question is how does pollution affect health.
• The study of newborns overcomes several difficulties in making the connection between pollution and health because the link between cause and effect is immediate.
• Selection bias is introduced given, air pollution which is is not randomly assigned and families with higher incomes or preferences for cleaner air are expected to sort into locations with better air quality.
• The study looked at the effect of E-ZPass in New Jersey and Pennsylvania on the health of infants.
• Used living near a Toll plaza (used as a proxy* for reduced air pollution) as the treatment.
• Used premature birth and low birth weight at the outcome.
• E-ZPass is an interesting policy experiment because while pollution control was an important reason for the state, most consumers used the pass to reduce overall travel time.
• Compares mothers within 2 km of a toll plaza to mothers who are between 2 km and 10 km from a toll plaza, but still within 3 km, of a major highway before and after the adoption of E-ZPass in New Jersey and Pennsylvania.
• Premature births tend to decrease 500 days after implementation of E-ZPass.
• Concluded that E-ZPass reduced both prematurity and low birth weights by 6.7-9.1% and 8.5-11.3% respectively
• The take away: policies intended to curb traffic congestion can have health benefits for local populations (in addition to often cited benefits in terms of costs).

Problems with staggered T

• In DID with staggered treatment, the analysis is composed of multiple 2X2 comparison around the time windows when any given unit is treated.
• If treatment effects are heterogeneous, then analyzing staggered treatments with "regular" DID biased results along with leads to any both Type I and Type II errors.
• Current literature in this sector has seen development new method versions which can deal with challenges around these concerns.

Other Applications

• Harjunen (2018) examined the West Metro extension in Helsinki and the effect on house prices, by comparing house prices in a close radius to the station to the prices outside.
• The date of was taken as 2009, when the concreate plans of the metro got initiated along with expectations from metro stations.
• Pekkarinen et al. (2009) found that the rollout implementation of policies related to education reform had positive impacts, with an elasticity of 0.3.
• A visualization or some sort of testing must show whether results are parallel before the treatment while asking whether is there anything else which could have happened to all groups.

These questions cover key aspects of evaluating the impact of the Millennium Villages Project (MVP). It focuses on understanding the 'big push' strategy, attribution challenges, the counterfactual, and the importance of rigorous evaluation methods.