Ordinary Least Squares (OLS) Estimation

• The document covers Ordinary Least Squares (OLS) estimation.
• It discusses assumptions, properties, and applications of OLS in linear regression models.

Simple Linear Regression Model

• The simple linear regression model is defined as (y_i = \beta_0 + \beta_1 x_i + u_i), for (i = 1, \dots, n).
• Here, (y_i) is the dependent variable, (x_i) is the independent variable, and (u_i) is the error term.
• (\beta_0) is the intercept, and (\beta_1) is the slope parameter.

OLS Estimators

• OLS estimators, denoted as (\hat{\beta}_0) and (\hat{\beta}_1), minimize the sum of squared residuals.

OLS Formulas

• The formula for (\hat{\beta}1) is (\hat{\beta}1 = \frac{\sum{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum{i=1}^{n} (x_i - \bar{x})^2})
• The formula for (\hat{\beta}_0) is (\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x})
• Here, (\bar{x}) and (\bar{y}) are the sample means of (x_i) and (y_i), respectively.

Fitted Values and Residuals

• Fitted values are given by (\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i).
• Residuals are given by (\hat{u}_i = y_i - \hat{y}_i).

Properties of OLS Statistics

• Sample average of residuals is zero: (\sum_{i=1}^{n} \hat{u}_i = 0).
• Sample covariance between regressors and residuals is zero: (\sum_{i=1}^{n} x_i \hat{u}_i = 0).
• The point ((\bar{x}, \bar{y})) is always on the OLS regression line.

Sum of Squares

• Total Sum of Squares (SST) is defined as (\sum_{i=1}^{n} (y_i - \bar{y})^2).
• Explained Sum of Squares (SSE) is defined as (\sum_{i=1}^{n} (\hat{y}_i - \bar{y})^2).
• Residual Sum of Squares (SSR) is defined as (\sum_{i=1}^{n} \hat{u}_i^2).

Decomposition of Total Variation

• SST = SSE + SSR
• This equation represents the decomposition of the total variation in the dependent variable.

R-squared

• R-squared ((R^2)) is defined as SSE/SST.
• It represents the fraction of the sample variation in y that is explained by x.
• (R^2) is also the squared correlation between (y_i) and (\hat{y}_i).
• (0 \leq R^2 \leq 1)

Assumptions for Unbiasedness of OLS

• Assumption SLR.1: Linear in Parameters: The population model is linear in parameters.
• Assumption SLR.2: Random Sampling: A random sample of size n is drawn from the population.
• Assumption SLR.3: Sample Variation in the Explanatory Variable: The sample values of (x_i) are not all the same.
• Assumption SLR.4: Zero Conditional Mean: (E(u|x) = 0).

Theorem 2.1: Unbiasedness of OLS

• Under assumptions SLR.1-SLR.4, (E(\hat{\beta}_1) = \beta_1) and (E(\hat{\beta}_0) = \beta_0).
• This means OLS estimators are unbiased estimators of the true parameters.

Variance of OLS Estimators

• (Var(\hat{\beta}1) = \frac{\sigma^2}{\sum{i=1}^{n} (x_i - \bar{x})^2})
• (Var(\hat{\beta}0) = \sigma^2 \left[ \frac{1}{n} + \frac{\bar{x}^2}{\sum{i=1}^{n} (x_i - \bar{x})^2} \right])
• Where (\sigma^2 = Var(u)) is the variance of the error term.

Assumption SLR.5: Homoskedasticity

• (Var(u|x) = \sigma^2) (constant variance)
• The error term has the same variance given any value of the explanatory variable.

Theorem 3.1: Sampling Variances of OLS Estimators

• Under assumptions SLR.1-SLR.5: The variance of (\hat{\beta}1) conditional on the sample values of x is (\frac{\sigma^2}{\sum{i=1}^{n} (x_i - \bar{x})^2})
• The variance of (\hat{\beta}0) conditional on the sample values of x is (\sigma^2 \left[ \frac{1}{n} + \frac{\bar{x}^2}{\sum{i=1}^{n} (x_i - \bar{x})^2} \right])

Properties of OLS Estimators: The Gauss-Markov Theorem

• Theorem 3.2: Gauss-Markov Theorem
• Under assumptions SLR.1-SLR.5, (\hat{\beta}_0) and (\hat{\beta}_1) are the Best Linear Unbiased Estimators (BLUE).
• 'Best' means minimum variance.

Estimating the Error Variance

• An unbiased estimator of (\sigma^2) is (\hat{\sigma}^2 = \frac{SSR}{n-2})
• This is also called the standard error of the regression (SER).

Standard Errors of OLS Estimators

• The standard error of (\hat{\beta}_1) is (se(\hat{\beta}1) = \frac{\hat{\sigma}}{\sqrt{\sum{i=1}^{n} (x_i - \bar{x})^2}}).
• The standard error of (\hat{\beta}_0) is (se(\hat{\beta}0) = \hat{\sigma} \sqrt{ \frac{1}{n} + \frac{\bar{x}^2}{\sum{i=1}^{n} (x_i - \bar{x})^2}}).

Assumption SLR.6: Normality of Error Term

• The population error u is independent of x and is normally distributed with mean 0 and variance (\sigma^2): (u \sim Normal(0, \sigma^2)).

Theorem 4.1: Normality of OLS Estimators

• Under assumptions SLR.1-SLR.6, (\hat{\beta}_0) and (\hat{\beta}_1) are normally distributed.

t-statistic

• t-statistic for (\beta_1) is (t = \frac{\hat{\beta}1 - \beta{1,0}}{se(\hat{\beta}_1)})
• Under H0: (\beta_1 = \beta_{1,0}), the t-statistic follows a t distribution with n-2 degrees of freedom.

Confidence Intervals

• A 95% confidence interval for (\beta_1) is (\hat{\beta}1 \pm t{n-2, 0.025} \cdot se(\hat{\beta}_1)).

Multiple Regression Model

• The multiple regression model is (y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_k x_k + u).
• (x_1, x_2, \dots, x_k) are the independent variables.

OLS Estimators in Multiple Regression

• OLS estimators minimize the sum of squared residuals: (\sum_{i=1}^{n} (y_i - \hat{\beta}_0 - \hat{\beta}1 x{i1} - \dots - \hat{\beta}k x{ik})^2).

Assumption MLR.1: Linear in Parameters

• The model is linear in its parameters.

Assumption MLR.2: Random Sampling

• A random sample of n observations is drawn from the population.

Assumption MLR.3: No Perfect Collinearity

• In the sample, none of the independent variables is constant, and there are no exact linear relationships among the independent variables.

Assumption MLR.4: Zero Conditional Mean

• (E(u|x_1, x_2, ..., x_k) = 0).

Theorem 3.1: Unbiasedness of OLS

• Under MLR.1-MLR.4, (E(\hat{\beta}_j) = \beta_j) for all (j = 0, 1, ..., k).

Assumption MLR.5: Homoskedasticity

• (Var(u|x_1, ..., x_k) = \sigma^2).

Theorem 3.2: Variances of OLS Estimators

• Under MLR.1-MLR.5, the variance of (\hat{\beta}_j) is (\frac{\sigma^2}{SST_j (1 - R_j^2)})
• (R_j^2) is the R-squared from regressing (x_j) on all other independent variables.

Multicollinearity

• If (R_j^2) is close to 1, (x_j) is highly correlated with other independent variables.
• High multicollinearity can inflate the variance of OLS estimators.

Omitted Variable Bias

• Occurs when a relevant variable is not included in the regression.

Variance Inflation Factor (VIF)

• (VIF_j = \frac{1}{1 - R_j^2})
• VIF quantifies the severity of multicollinearity.

Gauss-Markov Theorem for Multiple Regression

• Under MLR.1-MLR.5, the OLS estimators are BLUE.

Estimating (\sigma^2) in Multiple Regression

• (\hat{\sigma}^2 = \frac{SSR}{n - k - 1})
• Where (n - k - 1) is degrees of freedom.

R-Squared

• (R^2 = 1 - \frac{SSR}{SST})
• It measures the proportion of the total sample variation in y that is explained by all the independent variables together.

Adjusted R-Squared

• (\bar{R}^2 = 1 - \frac{SSR/(n-k-1)}{SST/(n-1)})
• Adjusted R-squared accounts for the number of independent variables in the model.
• (\bar{R}^2) can decrease when a variable is added to the model.

Assumption MLR.6: Normality of Error Term

• The population error u is independent of (x_1, ..., x_k) and is normally distributed with mean 0 and variance (\sigma^2): (u \sim N(0, \sigma^2)).

Theorem 4.1: Distribution of OLS Estimators

• Under MLR.1-MLR.6, OLS estimators are normally distributed.

t-Statistic in Multiple Regression

• (t = \frac{\hat{\beta}j - \beta{j,0}}{se(\hat{\beta}_j)})
• Follows a t distribution with (n - k - 1) degrees of freedom under the null hypothesis.

F-Statistic

• Used for testing multiple hypotheses.
• (F = \frac{(SSR_r - SSR_{ur})/q}{SSR_{ur}/(n-k-1)}), where subscripts r and ur denote restricted and unrestricted models.
• Follows an F distribution with (q, n-k-1) degrees of freedom under the null hypothesis.