7

• Empirical analysis focuses on moving from experimental data to observational data for estimating causal affects
• You will become familiar with common quasi-experimental causal inference methods
  • Instrumental Variables (IV)
  • Regression Discontinuity Design (RDD)
  • Difference-in-differences (DID)
• Designs based on controlling for observable differences are also covered

Homework 4 Outline

• Opens on Monday 3.2
• The deadline is in the last week of lectures (Feb 12)
• There is a practical homework assignment
• An exercise session with Lachlan on Thursday 6.2 is related

Homework 5 Outline

• Also known as the reading assignment
• Consists of a list of research papers
• Questions about the papers must be answered
• It will be due one week after the exam

Pre-lecture assignments

• Answers should show understanding with a few sentences each
• A good rule of thumb is to use a few sentences for each question
• Explanations of the concept or issue are important
• Short questions prepare for lectures and exams

Today's Lecture focuses on these topics:

• Limitations of Randomized experiments (RCTs)
• Observational data and the difficulty of making causal claims based on it
• Quasi-experiments
• Case study of studying neighborhood quasi-experimental effects.

Causal Inference Definitions

• Estimating variable T on outcome Y
• T = independent variable
• Y = the outcome variable

Causal Inference Examples

• Education (T) leads to earnings (Y)
• Marketing campaign (T) leads to sales (Y)
• Carbon tax (T) leads to emissions (Y)
• R&D subsidy (T) leads to innovation (Y)
• Fiscal stimulus (T) lead to unemployment (Y)

The Limits of RCTs

• Randomized experiments are often the best way to evaluate causal impacts of "treatments"
• Advantages:
  • Simple and transparent to understand results
  • Require fewer assumptions than alternative approaches
• Disadvantages: RCTs are not always feasible or desirable
  • Too costly
  • Unethical
  • Unhelpful for historical or understanding market level questions
• Observational data and clever design can still allow the researcher to study causal questions.

Ethical & Practical Limitations of RCTs

• Experiments should not knowingly harm anyone
• Can pose an issue when trying to study the effect of potentially harmful things
• The relevant time horizon may be very long
• Lengthy experiments may be needed when studying the role of a given education on future labor market outcomes throughout adult life
• Meaningful experiments are sometimes very expensive
• Policy and business mistakes are costly even though not experiments
• Hawthorne and John Henry Effects = the evaluation itself can push people to change their behavior
• Less of a problem with administrative data and long follow-up periods

Examples of ethical limitations for RCTs

• Political propaganda on political views or beliefs
• Fertility on women's wages
• Exposure to violence, war, or pollution on human capital development
• Introducing random variation in these treatments would be unethical
• Alternative: find some exogenous factor:
  • Introduces random variation in participation / exposure to war or violence or pollution.

Fundamental limitations of RCTs:

• Spillovers:
  • occur when the treatment also affects the control group
  • leads to issues inferring what would have happened without the treatment
• General equilibrium (GE) effects:
  • GE effects may be the main value of some treatments
  • RCTs never capture economy-wide GE effects
  • Some examples of measuring more limited spillovers with RCTs exist.
• Scarcity of potential observations
  • Some treatments affect entire countries or even the whole world and there will never be adequate designs for them

Observational data

• Researchers use observational data when experimental designs are infeasible
• Collected as part of normal functioning of society's institutions, and firms, including:
  • Data from administrative records
  • Surveys
  • Sales Data
  • Censuses
  • Apps
• Observational studies draw inferences from a sample of a population
  • The independent variable or "treatment" is not under the researcher's control
• In contrast with experiments, such as randomized controlled trials (RCTs)
  • Each the subject is randomly assigned to a treatment or a control group

Observational data and Correlations

• Correlations found in observational data should be viewed with suspicion
• Most correlations are not linked causally
• Variables endogenously chosen by the involved people contribute to the choice
• The FLEED data previously provided by Statistics Finland is from education level of the people in FOLK data, and the results are optimization by those people
• Economic theory explains people's decisions are not random on average
• Causal effects are challenging to estimate

People Optimize

• T= education and Y= Earnings
• In the potential outcomes model the treatment must be completely independent to measure the effect
• Violations of the condition are caused by a person choosing a level of education for themselves
• Economic theory predicts that choices are endogenous, and thus naïve correlations are misleading
• Selection bias must be kept in mind

Observational alternatives to experiments

• Selection on observables: occurs when treatment and control groups differ from each other w.r.t. observable characteristics
  • Multivariate regression and matching
• Selection on unobservables: occurs when treatment and control groups differ from each other in unobservable characteristics
  • can occur if something unexpected happened in an “almost random” manner
• "Structural Approaches" can offer useful alternatives

Natural or Quasi Experiments

• Broad term for many different situations requiring different types of research methods
  • IV
  • RDD
  • DID
• Researcher utilizes government policy or acts of nature that affect households in a way that resembles an experiment
  • Results in control and treatment groups
• Historical episodes sometimes provide observable "pseudo" random variation in treatment status
• Might be caused by law changes that affect some people, but not others

Quasi Experiments

• Studying neighborhood effects with a quasi-experimental study

Segregation in a model city

• Assuming two residential areas with a fixed supply of housing
  • Area 1: Central City - historic city center
  • Area 2: Suburb - far away with lower quality amenities and less services
• Assuming two types of housesholds
  • Type 1: Rich
  • Type 2: Poor
• With both working in the city center and disliking commuting

Where will the rich end up living?

• With these assumptions and a free market, the city will be segregated by income
• Prices will be higher in the attractive city center
• Rich live in the city center, and poor live in the suburbs
• Segregation results from income inequality and optimizing behavior
• Resulting neighborhood effects can range

Neighborhood effects

• The neighborhood an individual grows up in can determine their peer group, education, role models and aspirations
• Poor neighborhoods have less opportunities than affluent neighborhoods

Observational data

• If we wanted to estimate the effect of living next to high income families
• Observing any effects from segregation will be difficult due to selection problems

Focusing on a Moving Low-income family

• Focus on one family moving from low to high income areas
  • Neighborhood quality would increase
  • The children could have different role models and peers
• Will the children in the family benefit from moving next to high income families?

Housing market mechanism and selection bias

• If living in richer neighborhoods affects nbd quality and peer groups (T) and/or children's education and future wages?
• Is children growing up in rich neighborhoods and doing better just correlation?
• Is that due to optimization behavior by the parents or a causal effect?

Selection into neighborhoods

• The same resources that affect the choice of what neighborhood to live in affect the child outcome directly
• How can the link between living in a rich treatment neighborhood to outcomes be isolated?

Controlling for observable differences?

• May be possible to focus for observable differences to improve insight

Similar People

• This would mean compare people who are:
  • similar on observable and measurable characteristics,
  • have the same initial income, level of education etc.,
  • but live live in different quality neighborhoods
• If families are assumed to be similar, why did the families make different residential location choices?
• Low-income parents who make the effort to move to a higher quality nbd probably also use more of other resources in parenting - This assumes observably similar parents who remain in the poor neighborhood

Public housing demolition as a quasi-experiment

• What if we provided one low-income family the resources to move to the other residential area
  • Neighborhood quality would increase
  • The children would have different role models and peers
• Would the benefit if moved?

Chyn 2018 on Public Housing Demolitions in Chicago

• The study looks at demolitions in Chicago
• It focuses on the the long-Run Effects of Public Housing Demolition on Children
• This paper focuses on the long-run outcomes of children.
• It studies public housing demolitions in Chicago, which forced low-income households to relocate to less disadvantaged neighborhoods using housing vouchers.
• Specifically, it compares:
  • displaced to their peers who lived in nearby public housing
  • those that were not demolished
• Concludes that displaced children are more likely to:
  • be employed
  • earn more in young adulthood.
• Displaced children also have fewer violent crime arrests
• Children displaced at young ages have:
  • lower high school dropout rates.

Quasi-experiment

• Studies the case of Chicago where:
  • The housing authority reduced its stock of public housing during the 1990s
  • Authority targeted buildings with poor maintenance for demolition
  • Other buildings were nearby were left untouched
  • Residents selected for demolition received Section 8 housing vouchers
  • They were forced to relocate
• This policy created a treatment and a control group “naturally” or by accident
  • The housing group was not planning to divide residents for research purposes
  • Researcher was not involved in creating the groups

Chyn (2018 AER) Limitations

• The Chyn paper is a particular type of quasi-experiment
  • It can be analyzed exactly as if it were a randomized experiment

The Paper

• Considers what study gives insights different from related research by focusing on treatment.
• The kind of observational data that is used must be presented
• Must check if the groups really look like they are randomized
• Must look if pre-treatment covariates must be balanced across groups (balance tests)
• Main results should be checked
  • including Heterogeneity w.r.t sex and age etc.

Research Design

• Compares the the young adult outcomes of displaced and non-displaced children from the same public housing project
• Treatment = being displaced

Key assumption 1

• Demolition decision is unrelated tenants' characteristics for validity
• Important requirement: tenant selection mechanism did not allow households to self-select into buildings for this validity
• Occurs if households were assigned (as-good-as) randomly
• Caused by a surplus of waiting applicants where severe need for affordable housing means people chose the units they have

Key assumption 2

• Need for careful showing of households/children in control/treatment
• If they are similar observably/ nonobservably it is plaussible the researcher does not observe
  • Results need to be tested for balancing
• Demolition should have have no effects on nondemolished
• Prior research on the same demolitions show it is violated because of crime
• Results are be biased toward zero (underestimates)

Chyn study Results

• The point estimate for the treat-control difference is 0.066 - In the treated (displaced) group - The employment rate is 6.6 ppt higher
  • The study also showed households often ended up in "better" ones than was possible
• Heterogenous effects are present and measured in sub groups sorted by:
  • Sex
  • Demolition age

Discussion 1

• Aims to estimate family's moving from poor neighborhood success
• Both Chetty et al. (MTO paper) and Chyn find that younger kids benefit more

Validity

• Internal validity: Do we learn the true effect for the treated population?
• are free of selection bias?
• External validity: Can we extrapolate to other populations?