Regression Analysis Module 1

Regression Analysis

• Variance: The goal is to maximize SS Regression and minimize SS Error.
• Regression Coefficient (R2): Represents the proportion of variance in the dependent variable explained by independent variables, ranging from 0 (not explained) to 1 (explains all variability).
• Standardized Coefficients (Beta): Allows comparison of different predictors, generally ranging from -1 to +1.
• Semi-partial Correlation: Measures the unique contribution of a specific independent variable to the dependent variable, controlling for other predictors.

Identifying Unusual Scores

• Regression Diagnostics: Aim for an accurate relationship between predictors and outcomes.
• Outlier Scores: Small data points can have a large influence on the solution.
• Studentized Residual: Measures the difference between the predicted and observed scores.
• Leverage: Measures the influence of each data point on the regression line.
• Cook's Distance: Combines leverage and studentized residual to identify influential data points.

Assumptions of Regression

• Linearity: The relationship between predictors and outcomes should be linear.
• Normality: Residuals should be normally distributed.
• Homoscedasticity: Residuals should have constant variance.
• Independence: Observations should be independent.

Multiple Regression

• Standard Multiple Regression: Used when the combination of predictors explains significant variance in the outcome variable.
• Hierarchical Regression: Used to test theoretical models, entering predictors in a specific order based on theoretical importance.

Mediated Regression Analysis

• Direct Effects: The straight arrow from predictor to outcome variable.
• Indirect Effects: The arrow from predictor to mediator to outcome variable.
• Mediating Variables: Explain how predictor variables influence the outcome variable.
• Fully Mediated Effects: When the predictor variable influences the outcome variable only through the mediator.

Factors and Principal Components

• Principal Components Analysis (PCA): Reduces a large number of items to a smaller number of components, interested in all variance.
• Factor Analysis (FA): Interested in the common variance, what is shared among items (co-variance).
• Orthogonal Rotation: Produces independent components, maintaining independence between components.

Let me know if you'd like me to clarify anything!### Factor Analysis

• Deliberate mind wandering: enjoying mind wandering, thoughts wander on purpose, becoming absorbed in pleasant fantasy, and mind wandering to cope with boredom
• Types of mind wandering: deliberate and spontaneous

Principal Axis Solutions

• Uses a 2-factor solution
• FA eigenvalues are lower than PCA because PCA uses 100% of the variance, while FA only uses shared variance
• Scree plot is used to determine the number of factors to extract
• Factor matrix: all 8 items have strong loadings, but item 2 has a mix, which is not desirable
• Varimax rotation can improve the factor matrix

Complex Items

• Items that correlate more than 0.35 on more than one component in the rotated solution
• May be part of a general construct only and not explanatory when searching for independent dimensions
• Solution: exclude the item (if there is one), or if many, components are not independent; use an oblique solution to allow for correlations between components

Oblique Rotation

• Components are not independent; correlations between them
• Independent (varimax) - angle = 90 degrees
• Oblique (not independent) - each axis moves separately
• Not independent < 90 degrees

Pattern Matrix and Structure Matrix

• Pattern matrix: provides factor loadings that are unique (excludes shared variance)
• Structure matrix: factor loadings are higher than the pattern matrix and include shared variance (variance counted twice)
• Factor correlation matrix: gives the correlation between the two matrices

Writing Up an Oblique Solution

• Report total % of variance accounted for
• Describe factors using factor loadings
• Report correlation between factors produced

Assumptions of Analysis

• Bartlett's test of sphericity: determines whether there are factors/components in the correlation matrix
• Kaiser-Meyer-Olkin measure of sampling adequacy: describes the proportion of variance that might be described by underlying factors

Distribution of Items

• Scales suitable for PCA/FA: non-discriminating items (same score), extreme scores
• Bartlett's test: determines whether there are factors/components in the correlation matrix
• Kaiser-Meyer-Olkin measure of sampling adequacy: describes the proportion of variance that might be described by underlying factors

Strategy Analysis

• Inspect item distributions
• Correlation matrix: exclude items with correlations < 0.3 with at least one other
• Assess sampling adequacy
• Determine how many components to extract
• Remove variables that need to be discarded
• Perform rotations

Reliability Analysis

• Internal consistency of components: correlation between item and component
• Cronbach's alpha: measures the extent to which different items on a scale are measuring the same construct
• Good reliability: does not necessarily mean the instrument is valid

Testing Internal Consistency Factors

• Use items with high loading on each scale
• Reliability statistic: using Cronbach's alpha
• Item-total statistics: Cronbach's alpha if an item is deleted, corrected item-total correlation, and squared multiple correlation

Using Components in Other Analyses for Validity

• Can be saved and components/factors used in other analyses - added to data file
• Uses linear combination produced based on rotation used for each component/factor

Reporting Results

• Number of items included in final solution (as well as how many excluded)
• Assumptions
• % variance accounted for overall and how many components included in final solution
• If orthogonal rotation, % accounted for each component
• Use final solution (e.g., varimax), tell reader
• Final solution based on theoretical meaningfulness of outcome, not entirely statistics
• Table with all items on each component and loading
• Describe each component and name it
• Coefficient alpha (α) for each component

Factor Analysis

• Eigenvalue determines the percentage of variance accounted for in a factor
• An eigenvalue of 1 equals the proportion of variance of one item

Moderation and Mediation

• Moderator: a conditional/dependent effect where the influence of the independent variable (IV) on the dependent variable (DV) depends on the score of the moderator
• Example: years in education (IV), gender (moderator), and salary (DV)
• Mediator: a pathway or chain of events where the IV influences the mediator, which in turn influences the score on the DV
• Example: years in education (IV), beginning salary (mediator), and salary (DV)

Covariates

• Should be used in an analysis when you want to control for error to better explain the association between the IV and DV

Regression Basics

• Standard multiple regression predicts a dependent variable (DV) using two or more independent variables (IVs) simultaneously.
• A variable is a measurable characteristic that varies (by groups, individuals, or time).
• Dependent/Outcome Variable (DV) is the presumed effect in the analysis.
• Independent/Explanatory Variable (IV) is the presumed cause in an analysis.
• Control Variable/Covariate is a variable that is not studied but included in the model/analysis.

Regression Concepts

• Best Fitting Line: The most appropriate line showing the relationship between dependent and independent variables.
• Residual: Deviators from the fitted line (estimated value) to the observed values (data point).
• Error: Difference between the observed value and the true value (often unobserved).
• Unique Variance: Variability in a DV uniquely explained by specific IV(s) in multiple regression.
• Shared Variance: Variability in a DV explained by multiple IV(s) simultaneously in both multiple regression and Pearson's correlation.

General Linear Model (GLM)

• GLM: date = model + error (regression and ANOVA).

Regression Results

• Regression Coefficient R2: represents the proportion of the variance in the dependent variable that is explained by the independent variables in the model.
• Ranges from 0 (not explained) to 1 (explains all variability).
• Unstandardized coefficient: the slope of the regression line reflecting the change in the DV from one-unit change in the IV, whilst holding all other variables constant (B).
• Standardized coefficient: the slopes of the regression line expressed in standard deviation units (generally -1 to +1); making it comparable with other standardized coefficients.
• Semi-partial correlation (Part)sr2: Correlation between the predictor and outcome variable with variance shared between other predictors controlled in the predictor variable only.
• p-value of the model: It tests whether R2 is different from 0. A value less than 0.05 shows a statistically significant relationship.

Identifying Unusual Scores

• Influences the way the outcome of the analysis can be interpreted.
• Outlier Score: Studentised residual: unusual on IV.
• Discrepancy (most instable): unusual on IV and DV.
• Influential: unusual on IV: Mahanobalis and Leverage.

Assumption of Regression

• Linearity: The relationship between IV and the mean of DV is linear.
• Homoscedasticity: The variance of residual is the same for any of IV.
• Normality: For any fixed value of IV, DV is normally distributed.
• Multicollinearity: Associations between predictors (0) redundant (1) independent.

Selecting the "Best" Model

• Goal is to minimize the residual mean square (which maximizes R2) - by comparing regression models.

Hierarchical Regression Model

• Hypothesis model – we determine what happens based on theory.
• Entered into the model at different steps, based on theoretical importance or control.

Reporting

• R2 change = Squared semi-partial correlation.
• How much added variance in DV explained each step.
• Report significance of F change.
• Change for individual IVs not always significant.

Mediated Regression Analysis

• Mediating variables theoretically explain how the predictor variables influence the DV (outcome).
• The IV should precede the mediator in time, and the mediator should precede the DV.

Moderated Regression Analysis

• Influence of one IV on DV "changes" based on the score on the second IV.
• The moderator variable is the IV that influences the relationship between IV and DV, such as direction or strength.
• The IV is no longer independent; it is "conditional" on the moderator.

ANOVA Basics

• Are the means different?
• Definitions and Terms:
• T statistics: Tests whether two group means are significantly different.
• F statistics: The ratio of the model to its error.

Variability

• Between conditions: Explained by our model.
• Within conditions: Unexplained error.

Sum of Squares

• SS Total: Grand Mean.
• SS between: Variance explained by our model.
• SS within: Variance not explained by our model.

Degrees of Freedom

• df for SS between: k-1 (number of conditions/groups minus 1).
• df for SS within: N-k (Number of participants minus the number of groups).
• df SS total: N-1 (number of participants minus 1).

Omnibus Test

• Tests for an overall experimental effect – that the difference lies "somewhere".

ANOVA Designs

• Between groups: Two experimental conditions and different people are assigned to each condition.
• Repeated measures: Two experimental conditions and the same people take part in both conditions.
• Mixed ANOVA: Combination of repeated and independent factors.

Factorial ANOVA

• Factorial Designs can show interactions.
• The impact of one independent variable (IV) ignoring the presence of any other IV included in the design.
• Main effect: Influence of IV without regard for other IV's in the analysis.
• Interaction: The influence of one IV on the score of DV conditional (dependent) on the other independent variable.

Interaction Contrasts

• Evaluated effects of two IV's for different levels of the third IV.

Types of Interactions

• Disordinal Interaction: Effect of one IV, differs at the level of the second IV & direction or effect differs.
• Ordinal Interaction: Can interpret main effects (further analysis required on statistical interaction).

Regression Basics

• Standard multiple regression predicts a dependent variable (DV) using two or more independent variables (IVs) simultaneously.
• A variable is a measurable characteristic that varies (by groups, individuals, or time).
• Dependent/Outcome Variable (DV) is the presumed effect in the analysis.
• Independent/Explanatory Variable (IV) is the presumed cause in an analysis.
• Control Variable/Covariate is a variable that is not studied but included in the model/analysis.

Regression Concepts

• Best Fitting Line: The most appropriate line showing the relationship between dependent and independent variables.
• Residual: Deviators from the fitted line (estimated value) to the observed values (data point).
• Error: Difference between the observed value and the true value (often unobserved).
• Unique Variance: Variability in a DV uniquely explained by specific IV(s) in multiple regression.
• Shared Variance: Variability in a DV explained by multiple IV(s) simultaneously in both multiple regression and Pearson's correlation.

General Linear Model (GLM)

• GLM: date = model + error (regression and ANOVA).

Regression Results

• Regression Coefficient R2: represents the proportion of the variance in the dependent variable that is explained by the independent variables in the model.
• Ranges from 0 (not explained) to 1 (explains all variability).
• Unstandardized coefficient: the slope of the regression line reflecting the change in the DV from one-unit change in the IV, whilst holding all other variables constant (B).
• Standardized coefficient: the slopes of the regression line expressed in standard deviation units (generally -1 to +1); making it comparable with other standardized coefficients.
• Semi-partial correlation (Part)sr2: Correlation between the predictor and outcome variable with variance shared between other predictors controlled in the predictor variable only.
• p-value of the model: It tests whether R2 is different from 0. A value less than 0.05 shows a statistically significant relationship.

Identifying Unusual Scores

• Influences the way the outcome of the analysis can be interpreted.
• Outlier Score: Studentised residual: unusual on IV.
• Discrepancy (most instable): unusual on IV and DV.
• Influential: unusual on IV: Mahanobalis and Leverage.

Assumption of Regression

• Linearity: The relationship between IV and the mean of DV is linear.
• Homoscedasticity: The variance of residual is the same for any of IV.
• Normality: For any fixed value of IV, DV is normally distributed.
• Multicollinearity: Associations between predictors (0) redundant (1) independent.

Selecting the "Best" Model

• Goal is to minimize the residual mean square (which maximizes R2) - by comparing regression models.

Hierarchical Regression Model

• Hypothesis model – we determine what happens based on theory.
• Entered into the model at different steps, based on theoretical importance or control.

Reporting

• R2 change = Squared semi-partial correlation.
• How much added variance in DV explained each step.
• Report significance of F change.
• Change for individual IVs not always significant.

Mediated Regression Analysis

• Mediating variables theoretically explain how the predictor variables influence the DV (outcome).
• The IV should precede the mediator in time, and the mediator should precede the DV.

Moderated Regression Analysis

• Influence of one IV on DV "changes" based on the score on the second IV.
• The moderator variable is the IV that influences the relationship between IV and DV, such as direction or strength.
• The IV is no longer independent; it is "conditional" on the moderator.

ANOVA Basics

• Are the means different?
• Definitions and Terms:
• T statistics: Tests whether two group means are significantly different.
• F statistics: The ratio of the model to its error.

Variability

• Between conditions: Explained by our model.
• Within conditions: Unexplained error.

Sum of Squares

• SS Total: Grand Mean.
• SS between: Variance explained by our model.
• SS within: Variance not explained by our model.

Degrees of Freedom

• df for SS between: k-1 (number of conditions/groups minus 1).
• df for SS within: N-k (Number of participants minus the number of groups).
• df SS total: N-1 (number of participants minus 1).

Omnibus Test

• Tests for an overall experimental effect – that the difference lies "somewhere".

ANOVA Designs

• Between groups: Two experimental conditions and different people are assigned to each condition.
• Repeated measures: Two experimental conditions and the same people take part in both conditions.
• Mixed ANOVA: Combination of repeated and independent factors.

Factorial ANOVA

• Factorial Designs can show interactions.
• The impact of one independent variable (IV) ignoring the presence of any other IV included in the design.
• Main effect: Influence of IV without regard for other IV's in the analysis.
• Interaction: The influence of one IV on the score of DV conditional (dependent) on the other independent variable.

Interaction Contrasts

• Evaluated effects of two IV's for different levels of the third IV.

Types of Interactions

• Disordinal Interaction: Effect of one IV, differs at the level of the second IV & direction or effect differs.
• Ordinal Interaction: Can interpret main effects (further analysis required on statistical interaction).