Week 4: Simple Regression Model Basics

Week 4: Simple Regression Model (Part I)

• Model: A linear regression model is used to understand the relationship between a dependent variable (Y) and one or more independent variables (X).
• Assumptions of the Classic Linear Regression Model:
  • Assumption 1 (Linearity): The relationship between variables is linear in the parameters. This means that the dependent variable can be described as a straight line (or plane) when graphed against the independent variable(s).
  • Assumption 2 (Exogeneity): The independent variable(s) is not correlated with the error term. In simpler terms, the independent variable is unrelated to factors that influence the dependent variable through the error term.
  • Assumption 3 (Zero Conditional Mean): The expected value of the error term (u) is zero, given the value of the independent variable (X). Simply put, the error term's mean is zero. This ensures that factors not explained by X are not systematically linked to the independent variables.
  • Assumption 4 (Homoscedasticity): The variance of the error term is constant for all values of the independent variable(s). The spread of the dependent variable's values around the regression line is consistent across different values of the independent variables.
  • Assumption 5 (No Autocorrelation): There is no correlation between the error terms at different observations. This condition prevents the errors from being correlated. In simpler terms, the error at one observation does not influence the error at another observation.
  • Assumption 6 (Correct Specification): The regression model accurately represents the relationship between the variables.
  • Assumption 7 (Normality): The error terms follow a normal distribution with a mean of zero and constant variance (homoscedasticity).

Variances and Standard Errors of OLS Estimators

• The variances and standard errors of Ordinary Least Squares (OLS) estimators for b1 and b2 are essential for assessing the variability or uncertainty in these estimates.

Standard Error of Regression (SER)

• SER is a measure of the goodness of fit of the estimated regression line. It estimates the error in prediction. It is calculated using the formula: σ^2 = Σei^2 / (n - 2) and the standard error of the regression is its square root (ô = √σ^2)). This represents the average vertical distance between the observed data points and the fitted regression line.

Hypothesis Testing

• Testing hypotheses about the relationship between variables is a crucial aspect of regression analysis. Statistical tests are used to determine if the relationship is significant. A null hypothesis (e.g., no relationship between variables) is posited, and evaluated against an alternative hypothesis. Results are drawn based on either a confidence interval or significance test approach.

Test of Significance Approach

• The test of significance approach is a procedure for hypothesis testing, which depends on calculating a t-value and using critical values on a t-table.

One-Tailed Test

• A one-tailed test considers only a specific direction of the effect.

Two-Tailed Test

• A two-tailed test considers both directions of the effect.

Coefficient of Determination (r²)

• It measures how much of the variation in the dependent variable is explained by the regression model. It has values between 0 and 1. A higher value of r² indicates a better fit of the model.

Coefficient of Correlation (r)

• It measures the strength and direction of the linear relationship between the two variables. The value is between -1 and 1. A value of +1 or –1 signifies a perfect linear relationship, zero represents no relationship.

Normality Tests

• Various normality tests are utilized to verify that the errors or residuals in the regression model follow a normal distribution.
  • Histogram of residuals: Visualizes the distribution of residuals.
  • Normal Probability Plot (NPP): Plots the observed residuals against their expected values under a normal distribution. A straight line in the plot suggests a normal distribution.
  • Jarque-Bera Test: Tests the skewness and kurtosis of the residuals; a low p-value suggests deviations from normality.