Multiple Regression and OLS in Statistics

Multiple Regression

• When there are multiple predictor variables (X), we need to find a way to combine them to predict Y.
• The regression procedure extends to estimate b0, b1, b2, … bk, giving the best possible prediction of Y from all variables jointly.

Ordinary Least Squares Regression (OLS)

• OLS is a type of regression that estimates the best possible prediction of Y from all variables jointly.
• This type of regression is also referred to as multiple linear regression.

B-weights

• B-weights are partial slopes, telling us how much Y changes when one X changes by 1 unit, while holding other Xs constant.
• Each b-weight shows how much each X is related to Y when considering the interrelationships among Xs.

Beta (β) weights

• Beta weights are standardized b-weights, giving a measure of slope in standardized units.
• This allows for comparison between variables, but it's not a measure of importance.

Multiple Regression

• Multiple regression is an extension of bivariate regression, which is a special case of regression with a single predictor variable.
• Multiple regression allows us to account for the correlation between predictor variables.
• The intercept is the predicted score for scores of zero on each predictor variable.
• The slope (or partial slopes) represents the predicted value on Y for each 1 unit change on each X variable, while holding other predictor variables constant.

Coefficients and Variance

• Beta weights can be used to compare the association between different predictor variables and the outcome variable.
• R and R2 refer to the amount of variance explained in the outcome variable by the linear composite (i.e., the regression equation).

Multiple Linear Regression

• Used to understand the relationship between multiple predictor variables (X) and a response variable (Y)
• Model equation: Y = β0 + β1X1 + β2X2 + … + βpXp + ε

Model Components

• Y: Response variable
• Xj: jth predictor variable
• βj: Average effect on Y of a one-unit increase in Xj, holding all other predictors fixed
• ε: Error term

Estimating Coefficients

• β values are estimated using the least squares method, which minimizes the sum of squared residuals
• Matrix algebra is used to calculate the coefficients

Interpreting Coefficients

• Example: Exam score = 67.67 + 5.56(hours) - 0.60(prep exams)
• Each additional hour studied is associated with an average increase of 5.56 points in exam score, assuming prep exams are held constant
• Each additional prep exam taken is associated with an average decrease of 0.60 points in exam score, assuming hours studied are held constant

Correlation

• Measures the linear association between two variables, x and y
• Provides insight into how the variables move together
• Correlation coefficient (r) ranges from -1 to 1
• Perfect negative linear correlation: r = -1, as one variable increases, the other decreases
• No linear correlation: r = 0, variables are unrelated
• Perfect positive linear correlation: r = 1, as one variable increases, the other also increases

Example of Correlation

• Positive correlation between hours studied and exam scores: students who study more tend to earn higher exam scores

Regression

• Focuses on understanding how changes in the predictor variable (x) affect the response variable (y)
• Aims to find an equation that best describes this relationship
• Simple linear regression equation: ŷ = b0 + b1x
• Predicted value of the response variable: ŷ
• Y-intercept: b0, value of y when x is zero
• Regression coefficient: b1, average increase in y for a one-unit increase in x

Example of Regression

• Regression equation: Predicted exam score = 65.47 + 2.58 ⋅ (hours studied)
• Interpretation: a student who studies zero hours is expected to score 65.47, and each additional hour studied contributes an average increase of 2.58 points to the exam score

Key Difference

• Correlation does not imply causality, it merely describes the relationship
• Regression is founded on causality, aiming to predict outcomes based on predictor variables
• Correlation informs us about the strength and direction of the relationship, while regression helps us predict outcomes based on the predictor variable

Prediction and Forecasting

• Regression models enable prediction of future outcomes based on historical data
• Applications include finance (stock price prediction) and weather forecasting (temperature trend estimation)

Understanding Relationships

• Regression analysis helps understand the relationship between predictor variables and the response variable
• Coefficients identify which predictors have a significant impact on the outcome

Causal Inference

• Regression allows inference of causality, going beyond correlation analysis
• Controlling for other variables helps explore cause-and-effect relationships

Model Interpretation

• Coefficients provide insights into the strength and direction of relationships
• Interpretation helps decision-makers understand the impact of changes in predictors

Model Assessment

• Regression models can be evaluated using metrics like R-squared, adjusted R-squared, and root mean squared error (RMSE)
• These assessments guide model selection and improvement

Variable Selection

• Regression helps identify relevant predictors and exclude irrelevant ones
• Feature selection prevents overfitting and simplifies models

Correlation Coefficient and Hypothesis Testing

• Correlation coefficient (r) measures the strength and direction of the linear relationship between two variables (x and y).
• Reliability of the linear model depends on sample size (n).

Hypothesis Testing for Correlation Coefficient

• The goal is to determine if the population correlation coefficient (ρ) is significantly different from zero.
• Calculate sample correlation coefficient (r) from available data.

Conclusion Based on Test Results

• If (r) is significantly different from zero, the correlation is considered "significant".

• Conclusion: There is sufficient evidence to support a significant linear relationship between (x) and (y).

• Can use the regression line to model the relationship in the population.

• If (r) is not significantly different from zero, the correlation is considered "not significant".

• Conclusion: There is no significant linear relationship between (x) and (y).

• Cannot use the regression line to model the relationship in the population.

Practical Implications of Correlation Coefficient

• If (r) is significant and the scatter plot shows a clear linear trend, can use the regression line to predict (y) values for (x) values within the observed domain.
• If (r) is not significant or the scatter plot lacks a linear trend, the regression line should not be used for prediction outside the observed data range.
• Significance test helps decide whether the correlation is meaningful or due to chance.

Correlation Coefficient and Hypothesis Testing

• The sample correlation coefficient (r) measures the linear relationship between two variables in a sample.
• The population correlation coefficient is denoted by ρ (rho).
• The null hypothesis (H0) states that there is no linear relationship between X and Y in the population, i.e., ρ = 0.
• The alternative hypothesis (H1) states that there is a linear relationship between X and Y in the population, i.e., ρ ≠ 0.

Sampling Distribution of Correlation Coefficients

• When taking a random sample, the correlation coefficient (r) varies from one sample to another.
• The sample correlation coefficient (r) is not always zero, even if the population correlation coefficient (ρ) is zero.
• The sampling distribution of correlation coefficients is used to test the significance of the correlation.

Testing the Significance of R

• The significance of the correlation coefficient (r) is tested to determine if it is significantly different from zero.
• The test determines if the correlation in the sample is due to sampling error or if it reflects a non-zero correlation in the population.
• The concept of sampling distributions is used, similar to testing the significance of means.
• The test also examines the significance of each predictor (b weights) while controlling for all other predictors.

Regression Analysis

• Bivariate regression is a special case of regression that involves a single predictor variable.
• Multiple regression is an extension of bivariate regression, allowing for the addition of multiple predictor variables.
• The advantage of multiple regression is that it can account for the correlation between predictor variables.

Regression Coefficients

• The intercept represents the predicted score for scores of zero on each predictor variable.
• The slope (partial slope) represents the predicted change in Y for each 1-unit change in each X variable, while holding other predictor variables constant.

Measuring Associations

• Beta weights are used to compare the strength of association between different predictor variables and the outcome variable.

Evaluating Model Fit

• R and R² refer to the amount of variance explained in the outcome variable by the linear composite (i.e., the regression equation).