Simple Linear Regression Model

Introduction to Regression

• Regression analysis estimates relationships among variables, focusing on the connection between a dependent variable and one or more independent variables.
• Regression illustrates how the typical value of the dependent variable changes when an independent variable varies, while holding others constant.

Key Regression Concepts

• Dependent Variable (Response Variable): The variable that is predicted or explained, denoted as $y$.
• Independent Variables (Explanatory Variables, Predictors, Regressors, Features): Variables used to predict the dependent variable, denoted as $x_1, x_2, ..., x_p$.
• Linear Relationship: Presumes a straight-line relationship between independent and dependent variables.

Simple Linear Regression Model

• A basic model that uses one independent variable ($x$) and one dependent variable ($y$).
• Formula: $y = \beta_0 + \beta_1x + \epsilon$
• $\beta_0$: Intercept (the value of $y$ when $x = 0$).
• $\beta_1$: Slope (the change in $y$ for a one-unit change in $x$).
• $\epsilon$: Error term (the difference between observed and predicted values of $y$).

Ordinary Least Squares (OLS)

• OLS estimates parameters ($\beta_0$ and $\beta_1$) by minimizing the squared differences between observed and predicted values.

Loss Function

• The loss function measures the error in the model's predictions.
• Formula: $L(\beta_0, \beta_1) = \sum_{i=1}^{n}(y_i - (\beta_0 + \beta_1x_i))^2$
• $n$: Number of observations.
• $y_i$: Observed value of the dependent variable for the $i$-th observation.
• $x_i$: Observed value of the independent variable for the $i$-th observation.

Parameter Estimation

• Parameters are found by setting partial derivatives of the loss function with respect to $\beta_0$ and $\beta_1$ equal to zero.
• $\frac{\partial L}{\partial \beta_0} = -2\sum_{i=1}^{n}(y_i - (\beta_0 + \beta_1x_i)) = 0$
• $\frac{\partial L}{\partial \beta_1} = -2\sum_{i=1}^{n}x_i(y_i - (\beta_0 + \beta_1x_i)) = 0$
• Solutions for $\beta_1$ and $\beta_0$:
  • $\beta_1 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}$
  • $\beta_0 = \bar{y} - \beta_1\bar{x}$
• $\bar{x}$ and $\bar{y}$: Sample means of $x$ and $y$, respectively.

Multiple Linear Regression Model

• Extends simple linear regression to include multiple independent variables.
• Formula: $y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_px_p + \epsilon$
• $p$: Number of independent variables.
• $\beta_j$: Coefficient for the $j$-th independent variable, representing the change in $y$ for a one-unit change in $x_j$, holding all other variables constant.

OLS in Matrix Form

• The multiple linear regression model can be expressed in matrix form.
• Formula: $y = X\beta + \epsilon$
• $y$: $n \times 1$ vector of observed values of the dependent variable.
• $X$: $n \times (p+1)$ matrix of observed values of the independent variables, with a column of 1s for the intercept.
• $\beta$: $(p+1) \times 1$ vector of coefficients to be estimated.
• $\epsilon$: $n \times 1$ vector of error terms.

OLS Estimator

• The formula for the OLS estimator for $\beta$ is:
  • $\hat{\beta} = (X^TX)^{-1}X^Ty$

Assumptions of Linear Regression

• Linearity: Linear relationship between independent and dependent variables.
• Independence: Errors are independent of each other.
• Homoscedasticity: Errors have constant variance across all levels of the independent variables.
• Normality: Errors are normally distributed.
• No Multicollinearity: Independent variables are not highly correlated with each other.

Evaluation of Regression Models

• Evaluation assesses how well the model fits the data and predicts new data.

Key Metrics for Evaluation

• Mean Squared Error (MSE): Average of the squared differences between observed and predicted values.
  • $MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$
• Root Mean Squared Error (RMSE): Square root of the MSE.
  • $RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}$
• R-squared ($R^2$): Proportion of variance in the dependent variable predictable from the independent variables.
  • $R^2 = 1 - \frac{\sum_{i=1}^{n}(y_i - \hat{y}i)^2}{\sum{i=1}^{n}(y_i - \bar{y})^2}$
  • $R^2$ ranges from 0 to 1; higher values indicate a better fit.
• Adjusted R-squared: Modified $R^2$ that adjusts for the number of independent variables in the model.
  • $Adjusted \ R^2 = 1 - \frac{(1 - R^2)(n - 1)}{n - p - 1}$ -Adjusted $R^2$ is useful for comparing models with different numbers of independent variables.