Linear Regression Models PDF

## Interpretation of R Output for Linear Regression Models - **R output**: Includes coefficients, standard errors, t-values, p-values, residual standard error, R-squared, and F-statistic. - **Coefficients**: Estimated values of the regression parameters (β₀, β₁, ...). These are the intercept and sl...

## Interpretation of R Output for Linear Regression Models - **R output**: Includes coefficients, standard errors, t-values, p-values, residual standard error, R-squared, and F-statistic. - **Coefficients**: Estimated values of the regression parameters (β₀, β₁, ...). These are the intercept and slopes for each predictor variable. - **Standard Errors**: Measures of the variability of the coefficient estimates. - **t-values**: Test statistics for the hypothesis that each coefficient is zero. - **p-values**: Probabilities associated with the t-values, used to test the significance of each coefficient. - **Residual Standard Error**: An estimate of the standard deviation of the residuals, which measures the average distance that the observed values fall from the regression line. - **R-squared**: The proportion of variance in the dependent variable explained by the model. - **F-statistic**: A test statistic for the overall significance of the model, comparing the model with a null model (no predictors). ### Example | Coefficients | Estimate | Std. Error | t value | Pr(>|t|) | |---|---|---|---|---| | (Intercept) | -2.767 | 2.547 | -1.086 | 0.3029 | | x1 | 10.472 | 3.602 | 2.907 | 0.0156 * | - The intercept (β₀) is estimated as -2.767, and the slope (β₁) for x1 is 10.472. - The p-value for x1 is 0.0156, indicating it is statistically significant at the 5% level. ### Sum of Squared Residuals (SSR) - Measures the discrepancy between the observed data and the values predicted by the regression model. - Defined as: $SSR(β) = \sum_{i=1}^{n} (y_i - β'x_i)^2 = (y - Xβ)' (y - Xβ)$ - The goal of ordinary least squares (OLS) is to minimize the SSR, which leads to the best-fitting regression line. ### OLS Estimator - The solution to the minimization of the SSR. - Given by: $β_{OLS} = (X'X)^{-1}X'y$ - Provides the best linear unbiased estimator (BLUE) of the regression coefficients, assuming the Gauss-Markov conditions are met. ### Theorem of Gauss-Markov - States that under the following conditions: - **Linearity:** The model is linear in parameters. - **No multicollinearity:** The predictors are not perfectly correlated. - **Exogeneity:** The errors have zero mean and are uncorrelated with the predictors. - **Homoscedasticity:** The errors have constant variance. - **No autocorrelation:** The errors are uncorrelated with each other. - The OLS estimator is the best linear unbiased estimator (BLUE), meaning it has the smallest variance among all linear unbiased estimators. ### Hypothesis Testing, Confidence Interval, Significance Level, p-value - **Hypothesis Testing**: To determine whether a predictor variable has a statistically significant effect on the dependent variable. The null hypothesis (H₀) is typically that a coefficient is zero (no effect). - **Confidence Interval**: A range of values for a coefficient that is likely to contain the true value with a specified confidence level (e.g., 95%). For example: $β₁ ±1.96 · SE(β₁)$. - **Significance Level (α)**: The probability of rejecting the null hypothesis when it is true (Type I error). Common choices are 0.05 or 0.01. - **p-value**: The probability of observing a test statistic as extreme as the one computed, assuming the null hypothesis is true. If the p-value is less than α, the null hypothesis is rejected. ### R-squared, F Statistic - **R-squared**: Measures the proportion of variance in the dependent variable explained by the model. It ranges from 0 to 1, with higher values indicating a better fit. - **F Statistic**: Tests the overall significance of the model. It compares the model with a null model (no predictors). A large F-statistic indicates that the model is significant. ### Residual Standard Error - An estimate of the standard deviation of the residuals. - Measures the average distance that the observed values fall from the regression line. - Calculated as: $RSE = \sqrt{\frac{RSS}{n-p}}$ - The total sum of squares (TSS) measures the total variance in the dependent variable. - TSS is decomposed into the explained sum of squares (ESS) and the residual sum of squares (RSS): $TSS = ESS + RSS$ ### QQ Plots - (Quantile-quantile plots) are used to assess whether the residuals of a regression model are normally distributed. - The residuals are plotted against the quantiles of a normal distribution. - If the points fall approximately on a straight line, the residuals are normally distributed, which is an assumption of linear regression. ### Summary of Key Points - **R Output**: Provides estimates of coefficients, their standard errors, t-values, p-values, R-squared, and F-statistic. - **Sum of Squared Residuals**: Measures the discrepancy between observed and predicted values. - **OLS Estimator**: Minimizes SSR and provides the best linear unbiased estimates under Gauss-Markov conditions. - **Hypothesis Testing**: Uses p-values and confidence intervals to test the significance of predictors. - **R-squared and F Statistic**: Assess the overall fit and significance of the model. - **Residual Standard Error**: Measures the average deviation of observed values from the regression line. - **QQ Plots**: Used to check the normality of residuals.