Regression Analysis Concepts

• Multiple Regression Analysis is covered in the study notes
• These notes cover data analysis and business modelling
• Leila Tahmooresnejad is the instructor for this subject

Simple Linear Regression

• Y = b + b₁X
• Y represents the predicted value of Y
• b₀ is the estimate of βo, based on sample results
• b₁ is the estimate of β1, based on sample results

Excel Functions

• The intercept is calculated using: =INTERCEPT(known_y's,known_x's)
• The slope can be calculated using: =SLOPE(known_y's,known_x's)
• The trend is calculated =TREND(known_y's, known_x's, new_x's)

Errors

• Error = (Actual value) – (Predicted value)
• eᵢ = Yᵢ - Ŷᵢ

Coefficient of Determination

• The coefficient of determination is R²
• (R-squared) is a measure of the “fit” of the line to the data and it's value is between 0 and 1.
• SSR = Σ( Ŷ - Y)²
• SST = Σ(Y - Y)²
• R2 = SSR/SST
• The correlation coefficient is r
• Always between +1 and -1
• r = ±√r²

Linearity

• Linearity implies a linear trend is visible in the scatterplot
• Presence of no pattern is visible in the residual plot
• If the model is appropriate, then the residuals should appear to be randomly scattered about zero, with no apparent pattern

Performing Regression with Excel

• To perform a linear regression in excel navigate to: Data > DataAnalysis > Regression
• Select the labels box if the first row in the X and Y ranges includes the variable names
• Specify the location for the report output by clicking output range

Regression Output

• R2 is a coefficient of determination, a higher R2 close to 1 is desirable
• | r |, is the sample correlation coefficient.
• Significance F (p-value for overall model). A low value indicates a significant relationship between X and Y.
• T-stat and P-value are outputs in the Regression Statistics Table
• If the p-value is less than 0.05, reject the null hypothesis that the coefficient is zero

Topics Covered in the Study Notes

• Multiple Linear Regression
• Regression with Excel
• Building good Regression Models
• Multicollinearity
• Interactions

Multiple Linear Regression

• A linear regression model with more than one independent variable is a multiple linear regression model.
• Y = β₀ + β₁X₁ + β₂X₂ + ... + βₖXₖ + ε
• Y is the dependent variable.
• X₁,...,Xₖ are the independent variables
• β₀ is the intersect
• β₁,...,βₖ are the regression coefficients for the independent variables
• ε is the error term

Examples of Multiple Linear Regression Scenarios in Business

• Real estate pricing involves a dependent variable (house price) and independent variables (square footage, number of bedrooms, location, and age of the house).
• Predicting student performance uses a dependent variable (final exam score) and independent variables (hours spent studying, attendance, and prior academic history).
• Healthcare risk analysis considers a dependent variable (risk of developing heart disease) and independent variables (age, body mass index, blood pressure, and cholesterol levels).
• Measuring employee performance uses a dependent variable (employee performance metric) and independent variables (hours of training, years of experience, and job satisfaction)

Estimated Multiple Regression Equation

• Partial regression coefficients are estimated to use the model below:
• Ŷ = b₀ + b₁X₁ + b₂X₂ + ... + bₖXₖ
• The partial regression coefficients represent the expected change in the dependent variable
• When the associated independent variable is increased by one unit while the values of all other independent variables are held constant

Multiple Regression Model Prediction

• We can define a multiple regression model to predict a student's final exam score (Y)
• Three independent variables:
• Hours spent studying (X₁)
• Attendance percentage (X₂)
• Prior GPA (Χ₃)
• Ŷ = b ₀ + b ₁X₁ + b ₂X₂ + ... + b kX k (8.11)
• If the partial regression coefficient for X₁ is b ₁, it means that, for every one-hour increase in studying (while holding attendance and prior GPA constant), the student's final exam score is expected to increase by b₁ points.
• If the partial regression coefficient for X₂ is b₂, it means that, for every one-percentage-point increase in attendance (while holding studying and prior GPA constant), the student's final exam score is expected to increase by b₂ points.
• If the partial regression coefficient for X₃ is b₃, it means that, for every one-point increase in GPA (while holding studying and attendance constant), the student's final exam score is expected to increase by b₃ points.

ANOVA Testing

• Multiple R is the multiple correlation coefficient
• R Square is the coefficient of multiple determination
• ANOVA tests for significance of the entire model by computing an F-statistic for testing the hypothesis
• H₀: Β₁ = Β₂ = . . . = Βₖ = 0
• H₁: at least βᵢ is not 0

Hypotheses

• Multiple linear regression output provides the information to test hypotheses about each of the individual regression coefficients.
• If we reject the null hypothesis
• The slope associated with an independent variable is 0, then the independent variable i is significant and improves the ability of the model to better predict the dependent variable.
• If we cannot reject H₀, then that independent variable is not significant and probably should not be included in the model.

Performing Regression with Excel

• To perform a linear regression in excel navigate to: Data > DataAnalysis > Regression
• Input Y Range (with header)
• Input X Range (with header)
• Check Labels box
• The independent variables must be in contiguous columns
• So, columns of data may have to be manually moved around before applying the tool

Adjusted R²

• Is a modified version of R², which adjusts for the number of predictors in a regression model
• increases only if the additional predictors improve the model more than would be expected by chance
• To calculate it use the formula below: R²ₐ = 1 - [(n-1) / (n-k-1)] * (1 - R²)
• Where n = number of observations
• k = number of independent variables
• R²ₐ = adjusted R²

Systematic Model Building Approach

• Construct a model with all available independent variables.
• Check for the significance of the independent variables by examining the p-values.
• Identify the independent variable having the largest p-value that exceeds the chosen level of significance
• Remove the identified variable from the model and evaluate adjusted R²
• Don't remove all variables with p-values that exceed the variable at the same time, only one at a time
• Continue until all variables are significant.

Good Regression Models

• Variables should not be dropped at the same time and a structured approach is needed
• Independent variables must be significant
• Adding an independent variable to a regression model will always result in R² equal to or greater than the R² of the original model
• An increase in adjusted R² indicates that the model has improved

Multicollinearity

• Occurs when there are strong correlations among the independent variables, they can predict each other better than the dependent variable.
• When multicollinearity is present:
• It becomes difficult to isolate the effect of one independent variable on the dependent variable, as well as a difficulty to interpret regression coefficients because the signs of coefficients may be the opposite of what they should be.
• This can lead to misleading conclusions on the importance of certain variables because variables that are truly significant may appear insignificant due to the presence of multicollinearity.

Detecting/Addressing Multicollinearity

• Can be detected through correlation matrices and variance inflation factors (VIFs)
• High correlation coefficients or high VIF values (typically above 5 or 10) are indicators of multicollinearity
• Correlations exceeding ±0.7 may indicate multicollinearity
• ways to address it:
• Remove one or more highly correlated variables from the model (if possible, retain only the better predictors)
• Collect more data to reduce the effects of multicollinearity
• The variance inflation factor is a better indicator, but not computed in Excel

Model Development

• Identifying the best trend regression model often requires experimentation and trial and error.
• The independent variables selected should make sense in attempting to explain the dependent variable.
• Logic should guide your model development.

Avoiding Erroneous Models

• Weather data, while interesting, may not have a strong theoretical basis for directly predicting online purchase behavior.
• Weather conditions might impact certain types of businesses (e.g., outdoor retail) Additional variables increase R² and, therefore, help to explain a larger proportion of the variation.
• Good models are as simple as possible Even though a variable with a large p-value is not statistically significant, it could simply be the result of sampling error and a modeler might wish to keep it.

Overfitting

• Fitting a model too closely to the sample data at the risk of not fitting it well to the population in which we are interested
• R2-value will increase if we fit higher-order polynomial functions to the data. Overfitting can be mitigated by using good logic, intuition, theory, and parsimony.

Stepwise regression

• systematically adds or deletes independent variables.
• A forward stepwise procedure puts the most significant variable in first, adds the next variable that will improve the model the most.
• This type of regression begins with all the independent variables and deletes the least helpful (backward stepwise)

Regression

• Requires that you add categorical Variables, it is possible, but must code them numerically using dummy variables.
• Code as 0 and 1 for variables with 2 categories

Modelling Salary

• Y : β₀ + β₁Χ₁ + β₂X₂+ ε
• Y = salary
• x1 = age
• x2= MBA indicator

Interactions

• Occurs when the effect of one variable is dependent on another variable
• Y = β₀ + β₁Χ₁ + β₂X₂ + β₃X₃ + ε
• X3 = X₁ × X₂

Significant Interactions

• If b3 is statistically significant, this suggests that there is a moderating effect of employee experience on the relationship between job satisfaction and job performance.
• In practical terms, if b3 is positive and significant, it means that as employee experience increases, the positive relationship between job satisfaction and job performance becomes stronger.
• On the other hand, if b3 is negative, it indicates that higher levels of employee experience weaken the positive relationship between job satisfaction and performance.

Categorical Variables

• When a categorical variable has k > 2 levels, add k
• 1 additional variables to the model. Examples of this include:
• Salary and Education Level
• Employee Performance and Skill level
• Sales and customer satisfaction

General Pitfalls

• If the assumptions are not met, the statistical test may not be valid
• Correlation does not necessarily mean causation
• Multicollinearity makes interpreting coefficients problematic, but the model may still be good
• A t-test for the intercept (b0) may be ignored as this point is often outside the range of the model
• A linear relationship may not be the best relationship, even if the F test returns an acceptable value
• Even though a relationship is statistically significant it may not have any practical value

Explore key concepts in regression analysis, including dependent variables, error terms, and the coefficient of determination (R²). Understand its interpretations and the use of Excel functions for regression calculations. Learn about the application in real estate and employee performance models.