Regression Analysis Concepts

Quantitative Methods in Empirical Economic Geography

• This is a lecture on linear regression models, part III
• Lecturer: Christian Hundt
• Slides presented by Christian Hundt and Kerstin Nolte
• Location: Institute of Economic and Cultural Geography, Leibniz University Hannover

OLS Assumptions: The Gauss-Markov Theorem

• OLS stands for Ordinary Least Squares
• OLS yields consistent estimators for parameters β₀, β₁, ..., βₙ
• This is only true under certain assumptions.
• A consistent estimator converges to the true value of the parameter as sample size increases.

Assumptions in a Linear Regression Model

• Linear in parameters: A linear relationship exists between the dependent and independent variables.
• Random Sample: The sample must be representative of the population from which it was drawn.
• No perfect collinearity: No strong correlations between independent variables; one variable cannot perfectly predict another.
• Exogeneity of the predictors: Predictor variables are not correlated with the error term.
• Homoscedasticity: Error terms have constant variance.
• No autocorrelation: Error terms are not correlated with each other.

Checking for Linearity

• Use a residuals vs. fits plot to check for linearity.
• The residuals should be randomly scattered around a horizontal line at y = 0.

Logarithmic Transformation

• If a model is not linear in parameters, transform the variables.
• Logarithmic transformation is a common method to transform non-linear models into linear ones.
• Example: Cobb-Douglas production function can be transformed into linear form by using logarithms.

Random Sample of Size n

• If the sample is not random, it may be difficult to make inferences about the population.
• Collect data yourself.
• Use probability sampling methods.
• Probability sampling implies every member of the population has a nonzero chance of being picked in the sample.

No Perfect Collinearity

• Avoid perfect correlation between independent variables.
• Use variance inflation factor (VIF) to measure the strength of the correlation between variables in the model.
• A high VIF value indicates that an independent variable is correlated with other independent variables.
• High VIF values suggest that your model may be problematic; consider removing variables or using PCA.
• Variables that are perfectly correlated should be removed from the model.

Exogeneity of Predictors

• If a predictor variable(s) is correlated with the error term, it means that the predictor variable(s) is(are) endogenous.
• This includes, but is not limited to: misspecification, omitted variable, simultaneity.

Misspecification

• Could mean that the underlying data generating process is not correctly characterized or that some variables are not included in the regression.
• A missing variable in the regression will introduce bias to the other parameters, because it is correlated with at least one of the independent variables that were included in the model.

Omitted Variable

• If an important variable is missing in the model—this variable is correlated with one or more of the independent variables included in the model—the predictor variable coefficient estimates will also be biased.

Simultaneity

• An endogenous variable can be caused by other variables or it is a cause for other variables in a model.
• To fix this, instrument variables that are strongly correlated with the predictor variables but are not correlated with the dependent variable(s) can be used.

Homoscedasticity

• Examine the plot of the model's residuals vs. predicted values.
• If the residuals aren't randomly dispersed around 0, the variance isn't homoscedastic.
• Transforming the data—such as using logarithms—may fix heteroscedasticity. Calculating robust standard errors can also help.

No Autocorrelation

• Examine correlation plots among observations to determine if there is a trend.
• A violation of this assumption indicates that autocorrelation is present.
• This is true in the case of panel analysis and in the case of observations that are clustered (such as repeated measures from the same person or group).

Violations of the Assumptions

• A violation causes biased coefficients and biased standard errors.
• Address each issue individually.

Model Diagnostics and Strategy

• Examine the model assumptions after creating the model using diagnostics.
• Appropriately address violations. If assumptions are violated, re-evaluate the model's parameters and variables.
• If necessary, transform the data or use an alternative estimator (beyond OLS).

Creating a Regression Equation

• Start with a theory-driven variable selection: focus on variables with a clear theoretical relationship with the dependent variable.
• Use alternative methods (such as stepwise regression) only if theory isn't clear or if you have a lot of variables.
• A trade-off exists between including many variables and keeping the model simple and reliable.

Principal Component Analysis (PCA)

• Use PCA to combine multiple highly correlated variables to streamline data analysis.
• Creates a small set of uncorrelated variables from highly correlated ones.

Further Reading

• Wooldridge (2013), Introductory Econometrics. A Modern Approach (5th ed).
• Online resources for further details