Simple Linear Regression Model

Simple Linear Regression Model

• Regression aims to determine the effect of an independent variable (x) on a dependent variable (y).
• Linear relationships between variables are assumed.
• Regression allows for the estimation of y (a quantitative measure), given x; assume a quantitative measure for now.
• The formula for this is expressed as: y = β₀ + β₁x + ε
• β₀ represents the population parameter, y-intercept.
• β₁ represents the population parameter, slope.
• ε is error, accounting for unexplained variability within the linear relationship.
• The error term is a normally distributed random variable with a mean of zero and constant variance across all observations.

Estimated Simple Linear Regression

• Estimating population data requires taking a sample of the data.
• Formula for estimating is: ŷ = b₀ + b₁x
• ŷ is an estimate of the mean value of y given x, which is a point estimate.
• b₀ represents the sample parameter estimating the population parameter, y-intercept.
• b₁ represents the sample parameter estimating the population parameter, slope.
• b₁ can be interpreted as the unit increase in x and the average increase in y

Regression Process

• The regression process begins with determining the regression model y = β₀ + β₁x + ε.
• The next step is to determine the unknown parameters in β₀, β₁.
• Data is then sampled.
• The estimated regression equation of ŷ = b₀ + b₁x is constructed.
• The equations results in sample statistics of b₀, b₁.
• The values of b₀ and b₁ provide estimates of β₀ and β₁.

Possible Regression Lines in Simple Linear Regression

• In a positive linear relationship, the slope b₁ is positive.
• In a negative linear relationship, the slope b₁ is negative.
• In order to have no relationship, the slope b₁ is 0.

Regression Example

• Estimating the revenue of lightsabers selling with linear regression.
• The formula can be seen as ŷ = 150 + 3x, with x being the number of lightsabers sold.
• The estimated revenue after selling 10 lightsabers is $180, seeing that ŷ = 150 + 3x, ŷ = 150 + 3(10), ŷ = 150 + 30, ŷ = $180

Least Squares Method

• The least squares method is a procedure for using sample data to estimate the linear regression equation.
• This method minimizes the sum of squared errors.
• The Least Squares Equation is min Σ(Yi - y)² = min Σ(Y1 - bo – b121)².
• Within this equation, Yi is the observed value of the dependent variable for the ith observation, ŷ₁ is the predicted value of the dependent variable for the ith observation, n is the total number of observations

Extrapolation

• The experimental region is the range of independent variable values used for the regression.
• Extrapolation is predicting a variable outside the experimental region.
• Extrapolation may not have a regression model built with values outside the experimental region.
• There could also be a non-linear relationship further out.

Finding Sample Parameters

• Differential calculus can be used to determine the values of b₀ and b₁ that minimize expression.
• The slope equation is b₁ = Σ(xi - X) (Yi-Y) / Σ(xi - X)2
• The y-intercept equation is b₀ = y - b₁x
• Within this equation, Xi is the independent variable observation, yi is the dependent variable observation, X is the mean value for the independent variable, y is the mean value for the dependent variable, n is the total number of observations

Assessing the Fit of the Simple Linear Regression Model - Sum of Squares

• A way is needed to assess how well the model performs.
• Can assess through the Sum of Squares due to error (SSE), Total Sum of Squares (SST), and Sum of Squares due to regression (SSR)
• SSE = Σ(Yi – ŷi)²
• SST = Σ(уі – ӯ)²
• SSR = Σ(ŷi – y)²
• SST = SSE + SSR
• Where ӯ is the sample mean, yi is the actual sample value, and ŷi is the estimated sample value.

Assessing the Fit of the Simple Linear Regression Model - The Coefficient of Determination

• The Coefficient of Determination is commonly known as R2, where R is the correlation between x and y.
• This measures how much variability is explained by the model
• if R2 = 0.7, then 70% of the variability is explained by the model.
• The closer to 1 the better.
• A good range to have is 0.7 or above.
• 0 ≤ R2 ≤ 1.
• R2 = SSR/SST

Multiple Linear Regression Model

• A dependent variable (y) often relies on many variables, not a simple linear regression.
• Still assuming a linear relationship, but in higher dimensions, called a plane.
• The formula for multiple linear regression is: y = β₀ + β₁x1 + β₂x2 + … + βqxq + ε
• β₀, β1, β2,･･･, βq represent the population parameters
• ε is error, accounting for unexplained variability within the linear relationship
• The error term is a normally distributed random variable with a mean of zero and constant variance across all observations.

Estimated Multiple Linear Regression Model

• The estimated multiple linear regression model is: ŷ = b₀ + b₁x1 + b₂x2 + … + bqxq
• bo, b1, b2, …, bq are sample parameters estimating population parameters
• b₁ can be interpreted as the following when keeping everything else constant (i.e. x2, x3, …, xq ) for a 1 unit increase in x₁, y will increase by b₁ on average.

Least Squares Method for Multiple Linear Regression

• The least squares method is still used for multiple linear regression, but more complicated.
• The formula is min Σ (yi – y)² = min (Yi - bo - b121 -... - bqxq)² = min Σ εi²

Assessing Multiple Linear Regression Model

• Models are still assessed with the sum of squares and R².
• The formula becomes a bit more involved, but not something to worry about.

Conditions Necessary for Valid Inference in the Least Squares Linear Regression Model

• The goal of inference is to say something about the population based on the sample results.
• Conditions that must be met:
• For any combination of independent variables x1, x2, …, xq, the population of potential error terms ε is normally distributed with a mean of 0 and a constant variance.
  • In a plot of residuals, it will be centered around 0 with no discernable pattern.
• The values of ε are statistically independent, meaning each sample is independent.
• A large sample size (>30) can alleviate this.

Testing Individual Regression Parameters

• A few ways to make inferences about regression parameters:
• T-test
• P-value
• Confidence Interval

T-test

• A t-test for regression tests the hypothesis that the population parameter β₁ = 0 given what we calculated for bi
• H₀: β₁ = 0
• Ha: βi ≠ 0
• The test statistic is t = bi / Sbi
• b₁ is the regression coefficient
• S₁₁ is the standard deviation of regression coefficient

P-value

• A p-value is the probability of getting the result we obtained or more extreme assuming the null hypothesis is true.
• This value indicates if a regression coefficient (in turn the variable) is significant.
• A threshold of α = 0.05 is generally used for significance.
• If the p-value is LESS THAN 0.05 is significant, and this is the allowed false positive rate.
• The p-value is determined by using the t-distribution and calculating the area under the curve.

Confidence Interval

• A 100(1 – α)% confidence interval is an interval we are 100(1 – α)% confident the true population parameter is in that interval.
• For example, a 95% confidence interval for b₁ is (2,5). It can then be said that we are 95% confident the true population parameter is in the interval (2,5) using an open interval.
• The formal definition is that for confidence intervals constructed in this way, 95% of them will have the true population parameter in them.
• Confidence interval for βį is bi ± ta/2Sb₁ where ta/2 comes from the t-distribution.

Addressing Nonsignificant Independent Variables

• The p-value helps find significant variables.
• Thinking about the variables concerning a business question is important.
• The variable stays if the stakeholder wants it in the model.
• The variable stays if there is a domain expert aware of the relationship with the dependent variable.
• Make the decision to remove and build another model if neither is apparent.

Multicollinearity

• Multicollinearity refers to the high correlation between independent variables.
• The problem with multicollinearity is the regression model's coefficients are not accurate.
• Assess multicollinearity by looking at the actual correlation between variables and the variance inflation factor.
• (0.7 and above is bad) or variance inflation factor (VIF, 5 or more bad) is also bad

More Multicollinearity

• Always check for multicollinearity.
• Even if two independent variables are not highly correlated, a variable could be collinear with a combination of variables.
• A linear regression model can be made with the variable of concern as the dependent variable.
• R² above 0.5 is a concern.

Categorical Variables

• Categorical variables are as viable as quantitative numeric variables in linear regression.
• Dummy variables (0/1) are needed to represent the categorical variable.
• A 0 represents being not in a category and 1 represents being in the category.

Simple Example (2 categories)

• For example, the variable x that can either be 'Business student' or 'Not a business student'.
• x = 0 when a student is not a business student and x = 1 when a student is a business student.

Complex Example (More than 2 categories)

• A variable x that is the color of a Kyber Crystal has options 'Blue', 'Green', or 'Red'.
• In this example, there are three dummy variables.
• x₁: Is 1 if blue 0 otherwise
• x₂: Is 1 if green 0 otherwise
• 3: Is 1 if red 0 otherwise, but should be left off. For model variables, you will always leave off the last dummy variable since if you know 2 you know the other

Interpreting Dummy Variable Coefficient

• ŷ = bo + b₁x1 + b2x2 + … + bqxq
• Parameters bo, b1, b2, …, bq are sample parameters estimating population parameters
• b₁ (categorical): Keeping everything else constant (i.e. x2, x3, …, xq ) for x₁ being 1 (i.e. Having the category), y will increase by b₁ on average.

Non-Linear Relationships

• So far, has been linear regression, which assumes a linear relationship between the independent variables and dependent variables, but a non-linear relationship is also viable.

Quadratic Regression Models

• quadratic regression model, parabola!
• expressed as ŷ = bo + b₁x1 + b2x1²
• Same check of residuals applies.

Piecewise Linear Regression Model

• It is also known as a spline model.
• Idea is two linear regressions that are connected.
• This accounts for a changing slope.

Interaction Between Independent Variables

• A variable can be built as a multiplication of independent variables.
• two variables relate to each other and their combined effect has an effect on the dependent variable.
• Expressed as ŷ = bo + b₁x1 + b2x2 + b3X1X2
• Leads to many combinations.

Variable Selection Procedure

• With a lot of variables to choose from generally, it is difficult to choose the right ones. It is more of an art than a science.
• A starting point is significant variables.
• There are many stepwise selection methods:
• Forward Selection
• Start with no variables and continue to add variables. If a model gets better when added (with whatever metric you choose) then add it, if not do not add it.
• Backward Selection
• Start with all variables and continue to subtract variables. If model gets better when subtracted (with whatever metric you choose) then remove it, if not keep it.

Overfitting

• The goal of predictive modeling is to predict what is unkown. Overfitting is when your model performs well on the training data, but not on new data.
• Simpler models can avoid overfitting.
• Cross-validation with training and testing data can also avoid overfitting.
• This can be done for regression, but is usually not done.

Inference in Very Large Regression Samples

• When you have a very large data set, every variable will become significant.
• This causes a problem of variable selection, and requires you to dig deeper into the variables with your expertise.
• More data is always better, but consider the consequences and not just make a model blindly.

Model Selection

• With large data, model selection becomes more difficult.
• One can analyze through the sum of squares measures.
• One can look at AIC and BIC
• If there is a choice between a model with more variables or less variables with similar performance, pick the simpler model.

Predictions

• Given a regression model ŷ, a prediction can be with specific independent variables.
• Confidence intervals can also be found to give more accurate predictions.