5

• Causality involves understanding how one thing influences another.
  • Requires comparing counterfactual states ("how would Y change if we changed X?")
  • Only one state is observed.
• Key elements in a causal research question:
  • Outcome
  • Treatment
  • Counterfactual
  • Population
• A control group in experimental design helps infer what would have happened to the treatment group without the treatment.
• Selection bias occurs when control group is not comparable to the treatment group, leading to potential outcome differences.
  • E[y0|D = 0] ≠ E[y0|D = 1]
• Randomization eliminates selection bias, so the only difference between groups is treatment.
  • Differences in outcomes are due to the treatment.

Hypothesis Testing and Statistical Significance

• Key question: How likely is the difference between treatment and control groups due to chance?
  • Test the null hypothesis that the treatment had no effect.
• Learning Objectives focus on understanding the following concepts:
  • Point estimates
  • Standard errors
  • p-values
  • statistical significance
  • t-statistics
  • critical values
  • confidence intervals
  • false positives and negatives (Type I and II errors)
  • Use them for basic empirical results.
• Chattopadhyay and Duflo's 2004 study examined India's 1993 reform and policy outcomes.
  • Leaders invest more in infrastructure directly relevant to their own genders (e.g., women investing in drinking water).
• Data: 161 village councils ("Gram Panchayats" or GPs), 54 reserved for women leaders.

Point Estimate

• Average[y | D = 1] = 23.8
• Average[y | D = 0] = 14.7
• D = 1 denotes that GPs are reserved for female leader
• ӯ¹ – ӯº = P is the point estimate
• P more drinking facilities are built per village when a GP is led by a woman
• If GPs were randomly assigned into treatment and control groups, selection bias is deemed unlikely.
• Point estimates may differ from zero.
  • Female leaders are more likely to invest in drinking water.
  • 54 treatment GPs may invest more in drinking water for reasons unrelated to the gender of leader.
• The convention calls an estimate statistically significant if the likelihood of a chance finding is below 5%.

Simulating a Test Distribution

• An intuitive way to think about it is creating a distribution of "placebo" treatments.
• Procedure: split GPs randomly into "treatment" and "control" groups and then calculate averages.
• "placebo" assignments Dpl are made-up. Therefore, there is no impact.

P-Value

• P-value represents the probability of obtaining a result at least as extreme as the result actually observed under the null hypothesis.
• Here, the null hypothesis is zero treatment effect, i.e. Ho : E[y|D = 1] = E[y|D = 0].
• "2-sided” test: determines the likelihood of such a large deviation in absolute value from zero by chance.
  • Estimates are called "statistically significant"(reject null hypothesis) if the p-value is less than 5%.

Central Limit Theorem

• The central limit theorem: The sampling distribution of the sample mean of a large number of independent random variables is approximately Normal

Standard Errors

• Standard error is the standard deviation of a statistic.
• The statistic of interest here is the treatment effect estimate (difference between treatment and control group means).
• Variability in the treatment effect estimate is due to the random sampling randomness in treatment/control assignment.

Standard Errors: Calculation

• You can estimate the standard error for the difference in averages between two groups this way:
• ŜE(ӯ¹ – ӯº) = S(yi) √ 1/ n₁ + 1/n₀
• S(yi) is a sample standard deviation of y.
• n₁ and n₀ are the number of observations in the treatment and control groups.
• Alternative estimators for SEs exist, which each corresponds to different assumptions.
• Experiments yield more precise evidence when the outcome variable has less variation and the experiment is larger (higher n₁ and/or n₀)
• Earlier example: corresponding numbers are
  • ŜE(ӯ¹ – ӯº) = S(yi) √ 1/n₁ + 1/n₀ = 18.4 * √ 1/54 + 1/107 = 4.02*.
  • This is a close approximation for the standard deviation of the simulated test distribution.*

T-Statistic

• To denote, one must state the statistic of interest with к. One must also define the value using the null hypothesis using μ. Then the t-statistic: 
  • t(μ) = (K – μ)/SE(K).
• For treatment efforts, the hypothesis assumes = 0. Under that, the t-value using the average treatment effect is as following: 
  • t(0) = (K- μ)/SE(K) 
  • 1 – 0/ SE(1 - 0)
• the t-value is distributed, approximately, t to roughly N(0, 1), also referred to standard normal distribution
• The probability of drawing 2.26 or more from a standard Normal Distribution has a 1.19% chance of occuring
• It is more appropriate when the sample distribution is to calculate p-values 
• 2 sided p-value is 0.0238 

Critical Values

• A point in the test distribution has a corresponding p-value 
• in large samples, a t-statistic of 1.96 corresponds to a p-value of 0.05 in a 2 sided test. 
• A common rule-of-thumb is to call a result "statistically significant" if the point estimate is at least twice as large as its standard error 

Condfidence Intervals

• The relevant question is how large/small efforts can be ruled out 
• we can use confident intervals and 95% confidence intervals at that. 
  • [ẞ – 1.96 × SE, ẞ + 1.96 × SE]. 
  • 96 corresponds to a p-value of 0.05 in a 2-sided test where the statistic (e.g. average treatment effect) is distributed
• In an example, we had. If there's 9.1 then Se is 4.02. Therefore the answer is simply to find what the Cl is 95%
• Cls are graphically prevented/often presented. 

Misunderstandings

• Often times there is severe misunderstanding between "statistically significant" and the opposite to happen 
• treatment is incorrectly thought to have been "proven to be effective" when ***< .05 or "proven to have no effect" when **> .05.
• One simply must understand and interpret lots of research where statistical significance is used so one must abandon the whole concept 

Testing Errors

• False Positives: Saying that there is an error when it does not exist
  • Also referred to as a Type 1
• False Negative: Not finding an error when a result does not exist.
  • Aka called "Type 2" Error. 
• Power: The probability of finding when it occurs. 
• Statistical Significance: Its purpose is to avoid false positives 

This lesson covers interpreting test distribution simulations. It explains standard deviation, point estimates, and p-values. It emphasizes understanding statistical significance through simulations designed to test a null hypothesis.