Regression Analysis Basics

Regression Basics

• Standard multiple regression predicts a dependent variable (DV) using two or more independent variables (IVs) simultaneously.
• IVs have equal importance to explanation.
• Researchers not interested in associations between IVs.
• Key terms:
  • Variable: Measurable characteristic that varies (by groups, individuals, or time)
  • Dependent/Outcome Variable (DV): Presumed effect in the analysis
  • Independent/Explanatory Variable (IV): Presumed cause in an analysis
  • Control Variable/Covariate: Variables that are not studied but included in the model/analysis
  • Best Fitting Line: When plotting data, the most appropriate line showing the relationship between dependent and independent variables
  • Residual: Deviators from the fitted line (estimated value) to the observed values (data point)
  • Error: Difference between the observed value and the true value (often unobserved)
  • Unique Variance: Variability in a DV uniquely explained by specific IV(s) in multiple regression, distinct from Pearson's, where unique variance isn't assessed
  • Shared Variance: Variability in a DV, explained by multiple IV(s) simultaneously in both multiple regression and Pearson's correlation

Graphical Representation

• Total Variance, explained and error

Regression Results

• Regression Coefficient R2: represents the proportion of the variance in the dependent variable (the variable being predicted) that is explained by the independent variables (the predictors) in the model
• Ranges from 0 (not explained) to 1 (explains all variability)
• Unstandardized coefficient: the slope of the regression line reflecting the change in the DV from one-unit change in the IV, whilst holding all other variables constant (B)
• Standardized coefficient: the slopes of the regression line expressed in standard deviation units (generally -1 to +1); making it comparable with other standardized coefficients
• Semi-partial correlation (Part)sr2: Correlation between the predictor and outcome variable with variance shared between other predictors controlled in the predictor variable only
• p-value of the model: It tests whether R2 is different from 0. A value less than 0.05 shows a statistically significant relationship

Hierarchical Regression Model

• Hypothesis model – we determine what happens based on theory
• Entered into model at different steps, based on theoretical importance or control
• Associations between IVs important
• Most theoretically important variables entered at different steps
• Can test importance of different constructs

Statistical Regression Analysis (Stepwise)

• M – not theory (not recommended) – based on the size of the correlations
• Largest correlation is entered in first
• Atheoretical (statistically driven)

Mediated Regression Analysis (Cue Ball)

• Mediating variables theoretically explain how the predictor variables influence the DV (outcome)
• The IV should precede the mediator in time, and mediator should precede the DV
• Parallel mediator model: second mediator: can have two or more parallel mediators – need to be written for EACH indirect pathway association
• Mediated Regression Analysis (Baron & Kenny)
• 4 Steps:
  1. Path C: statistically significant association between IV and DV
  2. Path A: statistically significant association between IV and mediator
  3. Path B: statistically significant association between mediator and DV, after “controlling” for IV
  4. Path c’: association of IV & DV, after controlling for mediator – should be non-significant (full mediation) or reduced (partial mediation)

Moderating Regression Analysis

• Influence of one IV on DV “changes” based on score on second IV
• The moderator variable is the IV that influences the relationship between IV and DV such as direction or strength
• The IV is no longer independent; it is “conditional” on the moderator
• Moderator is a “conditional effect” = b3 tell us the condition
• Unconditional: The predictors each add variance to the explanation of the outcome variable, so each predictor is independent, so additive influence on the outcome
• Moderator effects mean that the IVs are not independent
• b3 = coefficient reflects the interaction between X*M eg. years in education * gender

ANOVA Basics

• Are the means different?
• Definitions and Terms:
  • T statistics: Tests whether two group means are significantly different
  • F statistics: the ratio of the model to its error
  • Variability
  • Between conditions: explained by our model
  • Within conditions: unexplained error
  • Sum of Squares
  • SS Total: Grand Mean
  • SS between: variance explained by our model
  • SS within: variance not explained by our model
  • Degrees of Freedom
  • df for SS between: k-1 (number of conditions/groups minus 1)
  • df for SS within: N-k (Number of participants minus number of groups)
  • df SS total N-1 (number of participants minus 1)
  • Omnibus test: tests for an overall experimental affect – that difference lies “somewhere”

Factorial ANOVA

• Factorial Designs can show interactions
• The impact of one independent variable (IV) ignoring the presence of any other IV included in the design
• Main effect: influence of IV without regard for other IV’s in the analysis
• Interaction: is the influence of one IV on score of DV conditional (dependent) on the other independent variable
• One level depends on the other level
• “The difference depends on..”

ANOVA Designs

• Between groups: two experimental conditions and different people are assigned to each condition (drug trial)
• AKA: “independent group”
• Repeated measures: two experimental conditions and the same people take part in both conditions (can control if individual error – separate error terms)
• Mixed ANOVA: combination of repeated and independent factors – participants 2 (reader group: dyslexia and control) × 2 (task difficulty: hard and easy): task difficulty repeated### Producing Independent Components
• Initial component matrix produces a general component, but it's not a good way to separate independent components.
• Factor rotation is used to reorganize the way variance is assigned to components, making them independent.
• There are two types of factor rotation: orthogonal (independent) and non-orthogonal (correlated) rotations.

Component Factor Rotations

• Orthogonal rotation:
  • Variance is extracted from individual items.
  • Components are independent from each other.
  • Naming and describing components/factors.
  • Interpreting outcome.

Representing 2 Components in 2D Space

• Loadings for each component extend from 1 to -1.
• Each item has a factor loading for each component, allowing representation in 2D space.
• Components are perpendicular (90° angle) and independent from each other.

Orthogonal Rotation

• Maintains independence of components.
• Items stay in the same place in 2D space.
• Principal axes are rotated to maximize the separation between the two groups.
• Varimax rotation is an example of orthogonal rotation.
• Variance accounted for in item-communality final (h2) is reported.

Naming Factors

• If replicating a solution, use previous names.
• Use a combination of items to describe the overall component.
• Avoid using the name of a single item.

Summary of PCA/FA

• PCA/FA are exploratory techniques that reduce a large number of items to smaller, more coherent dimensions.
• PCA/FA are based on a correlation/covariance matrix.
• Factor loadings are used to describe components descriptively.

Principal Components Analysis (PCA)

• PCA is interested in finding what components have in common.
• PCA explains the variance in each item using a few components.
• Communality is the percentage of variance explained in an item by the factor solution.

Factor Analysis (FA)

• FA is interested in finding the underlying components of a construct.
• FA is more theoretically driven.
• FA only uses shared variance (co-variance) between items.

Assumptions of Analysis

• Bartlett's test of sphericity determines if there are factors/components in the correlation matrix.
• Kaiser-Meyer Olkin measure of sampling adequacy describes the proportion of variance that might be described by underlying factors.

Distribution of Items

• Scales should be suitable for PCA/FA.
• Non-discriminating items (same score) and extreme scores can cause problems.
• Items should have a minimum of 3 response options, with 4 or more being better.

Strategy Analysis

• Inspect item distributions.
• Correlation matrix: exclude items with correlations < 0.3 with at least one other item.
• Assess sampling adequacy using the Kaiser-Meyer Olkin measure.