Linear Regression and Ordinary Least Squares (OLS)

• Linear Regression: A statistical method used to model the relationship between a dependent variable and one or more independent variables.
• The goal is to find the best-fitting linear relationship to predict the dependent variable's value based on the independent variable's values.

Ordinary Least Squares (OLS)

• OLS is a method for estimating the parameters in a linear regression model.
• It minimizes the sum of the squared differences between the observed values and the values predicted by the regression line.
• OLS provides the best linear unbiased estimators (BLUE) under certain assumptions.

Assumptions of OLS

• The error term has a zero population mean.
• The independent variables are uncorrelated with the error term.
• The error term has a constant variance (homoscedasticity).
• There is no autocorrelation among the errors.
• The error term is normally distributed.
• The regression model is linear in parameters.
• No perfect multicollinearity exists between independent variables.
• These assumptions are crucial for the validity and reliability of OLS estimates.

Hypothesis Testing

• Hypothesis testing is a statistical method used to determine whether there is enough evidence to reject a null hypothesis.
• In linear regression, hypothesis tests are used to determine the significance of individual coefficients or the overall model.
• Common tests include t-tests for individual coefficients and F-tests for overall significance.

T-test

• A t-test is used to determine if there is a statistically significant difference between the mean of a sample and a known value, or between the means of two samples.
• In regression, it tests whether an individual coefficient is significantly different from zero.
• The null hypothesis typically states that the coefficient is equal to zero.
• The t-statistic is calculated as the estimated coefficient divided by its standard error.

P-Value

• The p-value is the probability of observing a test statistic as extreme as, or more extreme than, the one computed if the null hypothesis is true.
• A small p-value (typically less than 0.05) suggests strong evidence against the null hypothesis.

Confidence Intervals

• A confidence interval provides a range of values within which the true population parameter is likely to fall.
• For regression coefficients, a confidence interval can be constructed around the estimated coefficient.
• If the confidence interval does not include zero, it suggests that the coefficient is statistically significant.

F-test

• An F-test is used to test the overall significance of a regression model.
• It tests whether all the coefficients in the model are simultaneously equal to zero.
• The null hypothesis states that all coefficients (except the intercept) are equal to zero.
• The F-statistic is calculated based on the ratio of explained variance to unexplained variance.

Model Specification

• Model specification involves choosing the appropriate variables and functional form for a regression model.
• It includes decisions about which independent variables to include, whether to include interaction terms, and the functional form of the relationship (e.g., linear, quadratic, logarithmic).
• Incorrect model specification can lead to biased and inconsistent estimates.

Omitted Variable Bias

• Omitted variable bias occurs when a relevant variable is excluded from the regression model.
• If the omitted variable is correlated with both the dependent variable and one or more included independent variables, the estimates of the included variables will be biased.

Irrelevant Variables

• Including irrelevant variables in the regression model can increase the standard errors of the estimated coefficients, making it harder to find statistically significant results.
• It can also reduce the overall fit of the model.

Functional Form

• Choosing the correct functional form is essential for accurate modeling.
• Linear relationships are not always appropriate, and nonlinear relationships may need to be modeled using quadratic, logarithmic, or other transformations.

Dummy Variables

• Dummy variables are used to represent categorical variables in a regression model.
• They take on values of 0 or 1 to indicate the presence or absence of a particular category.
• Dummy variables can be used to assess the impact of qualitative factors on the dependent variable.

Interaction Terms

• Interaction terms are created by multiplying two or more independent variables together.
• They allow the effect of one independent variable on the dependent variable to vary depending on the value of another independent variable.

Multicollinearity

• Multicollinearity occurs when two or more independent variables in a regression model are highly correlated.
• High multicollinearity can make it difficult to estimate the individual effects of the correlated variables.
• It can also inflate the standard errors of the coefficients, making it harder to find statistically significant results.

Detecting Multicollinearity

• Variance Inflation Factor (VIF): VIF measures how much the variance of an estimated regression coefficient increases if your predictors are correlated.
• Correlation Matrix: Check correlation coefficients between independent variables. High correlation (close to 1 or -1) indicates potential multicollinearity.

Addressing Multicollinearity

• Increase sample size: Larger samples can help reduce the standard errors of the coefficients.
• Drop one of the collinear variables: If two variables are highly correlated, consider dropping one of them from the model.
• Combine collinear variables: Create a new variable that is a combination of the collinear variables.

Heteroscedasticity

• Heteroscedasticity occurs when the variance of the error term is not constant across all observations.
• It violates one of the key assumptions of OLS regression.
• Heteroscedasticity does not cause bias in the coefficient estimates, but it does affect the efficiency of the estimates and can lead to incorrect inferences.

Detecting Heteroscedasticity

• Visual inspection of residuals: Plot residuals against predicted values or independent variables. A funnel shape suggests heteroscedasticity.
• Breusch-Pagan test: A statistical test for heteroscedasticity. It regresses the squared residuals on the independent variables and tests for a significant relationship.

Addressing Heteroscedasticity

• Weighted Least Squares (WLS): A regression technique that accounts for heteroscedasticity by weighting each observation differently based on its variance.
• Robust standard errors: These are standard errors that are corrected for heteroscedasticity. They can be used to make valid inferences even in the presence of heteroscedasticity.

Autocorrelation

• Autocorrelation occurs when the error terms in a regression model are correlated with each other.
• It is common in time series data, where observations are ordered chronologically.
• Autocorrelation violates one of the key assumptions of OLS regression.
• Autocorrelation does not cause bias in the coefficient estimates, but it does affect the efficiency of the estimates and can lead to incorrect inferences.

Detecting Autocorrelation

• Durbin-Watson test: A statistical test for first-order autocorrelation. It tests whether the error terms are correlated with their immediately preceding values.
• Visual inspection of residuals: Plot residuals against time. Patterns or trends suggest autocorrelation.

Addressing Autocorrelation

• Cochrane-Orcutt procedure: An iterative procedure that estimates the autocorrelation coefficient and transforms the data to remove autocorrelation.
• Newey-West standard errors: These are standard errors that are corrected for autocorrelation. They can be used to make valid inferences even in the presence of autocorrelation.