Multiple Regression in Survey Design and Analysis

Multiple Regression

• When there is more than one predictor (X) variable, multiple regression combines them to predict the outcome variable (Y)
• It extends the least squares procedure to estimate b0, b1, b2, ..., bk, giving the best possible prediction of Y from all variables jointly
• Also referred to as Ordinary Least Squares Regression (OLS)

B-Weights in Multiple Regression

• B-weights are partial slopes, each telling us how much change in predicted Y will occur for a change of 1 in that X, holding all other Xs constant
• This shows how much each X is related to Y, considering interrelationships among Xs

GRE-Q Example

• Examining the relationship between GRE-Q score and stats exam performance
• Considered predictors: GRE-Q score and attendance (every lecture or not)

Multiple Regression Output

• SPSS syntax: regression var= Stats_Exam GRE_Q Attendance /descriptive = def /dep= Stats_Exam /enter
• Output shows the correlation between the two predictor variables

Deconstructing the Regression Equation

• For a person scoring 0 on GRE-Q and 0 on attendance, the predicted score on Stats_Exam would be 36.130
• For every point scored on GRE_Q, the predicted score on stats_exam increases by 0.053, holding Attendance constant
• For every point "scored" on attendance, the predicted score on stats_exam increases by 12.157, holding GRE_Q constant

Standardised Weights

• Beta (β) weights are standardised b-weights, giving a measure of slope in standardised units
• This aids in comparison, but is not a measure of importance

Variance Explained

• R = 0.78 and R² = 0.608, meaning 60.8% of the variance in stats_exam is accounted for by the best linear composite of GRE_Q and Attendance

Regression Coefficients

• Are estimates of unknown parameters that describe the relationship between a predictor variable and the corresponding response variable.
• Used to predict the value of an unknown variable using a known variable, allowing for forecasting and inference.

Multiple Regression

• In multiple regression, we have more than one predictor (X) variable, and we need to combine them to predict the outcome variable (Y).
• The regression procedure expands seamlessly to estimate b0, b1, b2, …, bk to give us the best possible prediction of Y from all the variables jointly.

Interpreting Regression Coefficients

• The b-weights are the numbers by which we multiply each X to make a composite X with all the information from the separate Xs in it.
• b-weights are partial slopes, each telling us how much change in predicted Y there will be for a change of 1 in that X when all the other Xs are held constant.
• This gives us a clue as to how much each X is related to Y when considering the interrelationships among the Xs.

Example: Predicting Stats Exam Performance

• We are trying to understand the predictors of graduate statistics exam performance.
• We consider two predictors: 1) GRE_Q score and 2) Attendance (every lecture = 1, not every lecture = 0).
• We have already found that both higher GRE-Q scores and attending every lecture are positively related to graduate exam performance.

Regression Equation

• The regression equation predicts a score of 36.130 on Stats_Exam for a student who scored 0 on GRE_Q and 0 on attendance.
• For every point that a student scores on GRE_Q, their predicted score on Stats_Exam would increase by 0.053 when holding constant attendance.
• For every point that a student "scores" on attendance, their predicted score on Stats_Exam would increase by 12.157 when holding constant scores on GRE_Q.

Variance Explained

• R = 0.78, and R2 = 0.608, so 60.8% of the variance in Stats_Exam is accounted for by the best linear composite of GRE_Q and Attendance.
• This is a joint quantity and not the sum of the two separate correlations.

Standardized Weights (Beta Weights)

• Beta (β) weights are used for standardized solutions.
• They give a measure of slope in standardized units, aiding in comparison.
• IMPORTANCE OF MULTIPLE REGRESSION
• Multiple regression allows us to test the association of multiple predictors and a criterion variable.

Multiple Regression

• Multiple regression is used when there are more than one predictor (X) variables to combine them and predict the outcome variable (Y).
• It extends the least squares procedure to estimate b0, b1, b2… bk to give the best possible prediction of Y from all the variables jointly.
• This type of regression is also referred to as OLS, Ordinary Least Squares Regression.

Regression Equation

• The regression equation predicts the score on Stats_Exam based on GRE_Q and Attendance.
• For a person who scored 0 on GRE_Q and 0 on Attendance, the predicted score on Stats_Exam is 36.130.
• For every point increase in GRE_Q, the predicted score on Stats_Exam increases by 0.053, holding Attendance constant.
• For every point increase in Attendance, the predicted score on Stats_Exam increases by 12.157, holding GRE_Q constant.

Standardized Weights

• Beta (β) weights are used for standardized solutions.
• The intercept is zero and drops out of the equation.
• Beta weights give a measure of slope in standardized units, allowing for comparison between predictors.

Variance Explained

• R = 0.78, and R2 = 0.608, indicating that 60.8% of the variance in Stats_Exam is accounted for by the best linear composite of GRE_Q and Attendance.
• This is a joint quantity, not the sum of the two separate correlations.

B-Weights in Multiple Regression

• B-weights are partial slopes, each indicating how much change in predicted Y there will be for a change of 1 in that X, when all other Xs are held constant.
• This gives a clue as to how much each X is related to Y when considering the interrelationships among the Xs.

Predictors of Exam Performance

• GRE_Q score and Attendance are both positively related to graduate statistics exam performance.
• Regression allows us to test whether individual predictors are associated with the criterion when we account for overlap between the predictor variables themselves.

Assumptions of Multiple Regression

• Multiple regression assumes that variables are measured on a continuous scale.
• The criterion (dependent variable) must be continuous; if it's categorical or dichotomous, another type of analysis is needed, such as logistic regression.
• Categorical predictor variables can be used, but the criterion must still be continuous.

Multiple Regression

• Multiple regression is used to predict a continuous outcome variable (criterion) using multiple predictor variables
• In multiple regression, we have more than one predictor (X) variable, and we need to combine them to predict the outcome variable (Y)

Dummy Coding

• Dichotomous variables can be coded as 0 and 1, where 0 represents one group and 1 represents the other group

Regression Equation

• The regression equation is used to predict the outcome variable (Y) using multiple predictor variables (X)
• The equation is: Y = b0 + b1X1 + b2X2 + … + bkXk
• b-weights are the numbers by which we multiply each X to make a composite X with all the information from the separate Xs in it
• b-weights are partial slopes, telling us how much change in predicted Y there will be for a change of 1 in that X when all the other Xs are held constant

Example: Predicting Exam Performance

• The example uses two predictor variables: GRE_Q score and Attendance
• For every point that a student scores on GRE_Q, their predicted score on Stats_Exam would increase by 0.053 when holding constant Attendance
• For every point that a student "scores" on attendance (i.e., attends every lecture), their predicted score on Stats_Exam would increase by 12.157 when holding constant scores on GRE_Q

Standardised Weights

• Beta (β) weights are used to give a measure of slope in standardised units
• Beta weights aid in comparison, but are not a measure of importance
• Beta weights are used to compare the relative importance of each predictor variable

Variance Explained

• R = 0.78, and R2 = 0.608, which means that 60.8% of the variance in Stats_Exam is accounted for by the best linear composite of GRE_Q and Attendance

Assumptions of Multiple Regression

• Multiple regression assumes that the variables are measured on a continuous scale
• The criterion variable must be continuous, but categorical predictor variables can be used

Unstandardised Coefficients

• Unstandardised coefficients, also known as b coefficients, are the coefficients of the independent variables in a multiple regression equation.

Key Characteristics

• Measured in the units of the dependent variable (Y)
• Interpreted in terms of the original units of measurement
• Can be used to compare the effects of different independent variables on the dependent variable
• Can be used to make predictions of the dependent variable for specific values of the independent variables

Interpretation

• A positive unstandardised coefficient indicates a positive relationship between the independent variable and the dependent variable
• A negative unstandardised coefficient indicates a negative relationship between the independent variable and the dependent variable
• A coefficient of 0 indicates no relationship between the independent variable and the dependent variable

Example Application

• In the regression equation Y = 2X + 3Z + ε, the unstandardised coefficient for X is 2, indicating a 2-unit increase in Y for every one-unit increase in X, while holding Z constant.
• In the same equation, the unstandardised coefficient for Z is 3, indicating a 3-unit increase in Y for every one-unit increase in Z, while holding X constant.

B Weights

• Represent the relationship between an independent variable (predictor) and the dependent variable (outcome)
• Are not standardized and depend on the original units of measurement of the variables
• Can be useful for understanding the direction and magnitude of the effect of each predictor variable on the outcome
• Challenging to compare across different predictors because they are in different units

Beta Weights (β)

• Also known as standardized regression coefficients, derived from B weights
• Standardized by converting both predictor and outcome variables to z-scores (standard deviations from the mean)
• Represent the change in the outcome variable (in standard deviations) when the predictor variable increases by one standard deviation, while other variables are held constant
• Allow for cross-comparisons across different predictors because they are standardized
• Equal to the correlation coefficient between the predictor and the outcome when there is only one predictor
• Provide a clearer understanding of the relative impact of predictors on the outcome
• Focus on Beta weights for a clearer understanding of the relative impact of predictors on the outcome

Regression Analysis

• B weights (regression coefficients) represent the relationship between an independent variable (predictor) and the dependent variable (outcome).
• B weights are not standardized and depend on the original units of measurement of the variables.
• They are useful for understanding the direction and magnitude of the effect of each predictor variable on the outcome.

Standardized Regression Coefficients (Beta Weights)

• Beta weights (β) are derived from B weights by standardizing both predictor and outcome variables to z-scores.
• β represents the change in the outcome variable (in standard deviations) when the predictor variable increases by one standard deviation, while other variables are held constant.
• Beta weights are standardized, making it easier to compare the strength of predictions across different predictors.

Multiple Correlation Coefficient (R)

• R represents the correlation between the observed values of the response variable and the predicted values made by the regression model.
• R measures how well the model fits the actual data points.
• R ranges from -1 to 1, with:
  • Positive R indicating a positive linear relationship between predictors and the response.
  • Negative R indicating a negative linear relationship.
  • R = 0 indicating no linear relationship.
  • R = 1 or -1 indicating a perfect linear relationship.

Coefficient of Determination (R-squared)

• R² represents the proportion of the variance in the response variable that can be explained by the predictor variables in the regression model.
• R² ranges from 0 to 1, with:
  • R² = 0 indicating the model explains none of the variance (poor fit).
  • R² = 1 indicating the model explains all the variance (perfect fit).
• Interpretation: For example, if R² = 0.956, 95.6% of the variation in exam scores can be explained by the predictors (e.g., study hours and current grade).

Adjusted R-squared

• Adjusted R-squared accounts for the number of predictors in the model.
• It penalizes adding unnecessary predictors.
• Higher adjusted R-squared indicates a better model fit, adjusted for complexity.