Comparing Groups

• This module builds on first-year knowledge of comparing groups, particularly using T-tests
• The fundamental goal of comparing groups is to explain differences in data

Understanding Variability in Data

• Measuring different individuals on a test, data points are scattered on the X-axis, the primary focus is on variability on the Y-axis
• When comparing groups, the aim is to determine if a factor influences variability

Grouping Individuals for Analysis

• Groups may be formed quasi-experimentally based on pre-existing characteristics, or experimentally with random assignment
• The research question considers if grouping individuals help explain variability in the measured variable

Key Research Questions in Group Comparisons

• Consider if the grouping makes sense and reflects meaningful divisions
• Consider practical significance, justifying the effort if an intervention is time-consuming
• The magnitude of a difference determines practical importance, is the training worth the effort?
• Differences can exist between groups or between conditions

Null Hypothesis Significance Testing (NHST)

• A foundational concept in testing differences between groups
• Developed by Egon Pearson and Jerzy Neyman
• The Null Hypothesis (Ho) assumes no difference between conditions
• The Alternative Hypothesis (H1) assumes a real difference exists

Understanding P-Values

• Statistical analyses estimate the probability (p-value) of observing a difference if the null hypothesis were true
• Smaller p-values = Lower probability that the observed difference is due to chance
• Larger p-values = More likely that differences occurred randomly
• Conventionally, p < 0.05 is considered statistically significant

T-Tests: Comparing Two Means

• Used for comparing means between two groups

Independent Samples T-Test

• Used when comparing two separate, independent groups

Paired Samples T-Test

• Used when comparing the same participants under two conditions

One-Sample T-Test

• Compares a sample mean to a known population mean, less relevant in this context

Moving Beyond Two Groups: ANOVA (Analysis of Variance)

• Use ANOVA when presented with more than two groups, T-tests become insufficient

One-Way Between-Subjects ANOVA

• Compares means across more than two groups
• Helps determine whether at least one group differs significantly from the others
• If ANOVA finds a significant difference, post-hoc tests are needed to determine which groups differ

Practical Application in SPSS

• This module includes computer lab tutorials to conduct T-tests and ANOVA in SPSS
• SPSS tutorials provide practical experience in statistical analysis
• Comparing groups helps to explain variability in data
• Grouping individuals should be meaningful and justified
• Null Hypothesis Significance Testing (NHST) tests whether observed differences are statistically significant
• P-values indicate the likelihood of results occurring under the null hypothesis
• T-tests compare means between two groups, while ANOVA handles multiple groups

Overview

• Module focuses on T-tests for comparing two groups
• T-tests can be independent (between-group) or repeated measures (within-group)
• Goal is to determine if two means are significantly different

T-Statistic and Hypothesis Testing

• The T-statistic compares means and tests whether they are meaningfully different
• The Null Hypothesis (Ho) assumes no difference between the two groups
• The Alternative Hypothesis (H1) assumes a real difference exists
• Outcome of a T-test generates a p-value, indicating the probability the observed difference occurred by chance under Ho
• A p-value indicates the probability of no effect, not the probability of an effect

Types of T-Tests

Independent Samples T-Test (Between-Groups)

• Definition: Compares two separate groups with no overlap
• Each participant contributes one data point to the analysis

Repeated Measures T-Test (Within-Groups)

• Definition: Compares two conditions using the same individuals, measuring them before and after
• Each participant contributes two data points (one per condition)
• Advantage: Controls for individual differences because participants are their own control

Understanding Hypothesis Testing in T-Tests

Null vs Alternative Hypotheses

• Ho (Null Hypothesis): The population means are equal
• Any difference is due to random chance
• H1 (Alternative Hypothesis): The population means are not equal or there is a true difference between conditions

Understanding the P-Value

• The p-value represents the probability of obtaining the observed results if Ho is true
• Small p-values suggest rejecting Ho, implying a statistically significant difference

Sampling and Variability

Sampling Challenges

• We do not measure the entire population, only relying on samples and attempt to generalise results

Standard Error

• Measures random variability in sample means
• Determines if observed differences are larger than expected by chance

Conceptual Understanding of T-Tests

• Calculate the sample means for both groups
• Compare the difference between sample means
• Use Standard Error to determine if the difference is meaningful
• Generate a T-statistic and P-value to evaluate statistical significance
• Make a conclusion about whether to accept/reject Ho

T-Test Assumptions

Assumptions for Independent Samples T-Test

• Level of Measurement: the dependent variable must be interval or ratio, participants must be randomly selected
• Data should follow a normal distribution
• Both groups should have equal variance which can be checked using Levene's Test

Assumptions for Repeated Measures T-Test

• The dependent variable must be interval or ratio
• Participants must be randomly selected
• The differences between the repeated measures should be normally distributed
• Equal variance is not a concern because the same participants are tested in both conditions

Example: Exercise and Mood Study

• Research Question: Does exercise level affect mood?

Repeated Measures T-Test Example

• Independent Variable: Level of exercise (low vs. medium)
• Dependent Variable: Mood score (rated 1-10, higher = better mood)
• Same participants experience both conditions (low and medium exercise)
• Participants' mood is recorded after each exercise session

Results

• Low Exercise Mean Mood Score = 2.5 and Medium Exercise Mean Mood Score = 7.7, resulting in a difference of -5.2
• The p-value = 0.001, which is highly significant
• Counterbalancing is required to reduce order effects

Independent Samples T-Test Example

• Two separate groups (one does low exercise, one does medium), participants only ever experience one condition

Results

• Low Exercise Mean Mood Score = 1.6 and Medium Exercise Mean Mood Score = 8.2
• Levene's Test for Equal Variance is not significant because variances are equal

Limitations of T-Tests

• T-tests can only compare two groups
• If there are more than two conditions, ANOVA is needed

Issue of Multiple Comparisons

• Running multiple T-tests increases the risk of Type I errors (false positives)
• ANOVA allows comparison of multiple groups simultaneously

Key Takeaways

• Independent Samples T-Test: Compares two separate groups
• Repeated Measures T-Test: Compares the same participants across conditions
• Null Hypothesis (Ho): No difference between groups
• Alternative Hypothesis (H1): A true difference exists
• P-Value: Probability that results occurred by chance under Ho
• Levene's Test: Checks for equal variance for Independent T-tests
• T-Tests are Limited, if more than two groups use ANOVA instead

Introduction: Moving Beyond T-Tests

• Use One-Way ANOVA (Analysis of Variance) is used when an independent variable has three or more levels
• Examples: Comparing low, medium, and high exercise levels on mood or evaluating treatment effects before, immediately after, and at a follow-up

Problem: Type I and Type II Errors

• Type I Error (False Positive): Rejecting the null hypothesis when it is actually true
• Type II Error (False Negative): Failing to reject the null hypothesis when it is actually false

The Family-Wise Error Rate Problem

• Each T-test has a 5% chance of making a Type I error
• If multiple T-tests are used, the chance of making at least one Type I error increases dramatically

Solution: One-Way ANOVA

• Instead of running multiple T-tests, ANOVA allows to compare all groups at once
• Controls for error inflation and ensures a more reliable analysis

Understanding the One-Way ANOVA

Null and Alternative Hypotheses

• Null Hypothesis (Ho): All group means are equal. Example: μ₁ = μ2 = μ3
• Alternative Hypothesis (H1): At least one group mean is different. It does not specify which groups are different

ANOVA and the F-Statistic

• ANOVA produces an F-statistic (F-ratio) where F=Variance between groups/Variance within groups
• If F is large, there is a higher chance of a significant difference
• If F is close to 1, there is no significant effect as variability within groups is similar to the variability between groups
• ANOVA is an Omnibus Test and only tells us if a difference exists, but not where.
• If ANOVA is significant, post hoc tests are required to find which groups differ

How ANOVA Works

• Any dataset has a total amount of variability (Total Sum of Squares, SST)
• ANOVA breaks it down into Between-Groups Variability and Within-Groups Variability
• F-Statistic compares between-group variance to within-group variance
• If F is significant, at least one group differs and post hoc tests are run to identify which groups are different

Exercise Intensity and Mood

• Research Question: Does exercise level (low, medium, high) affect mood?
• Independent Variable (IV): Exercise intensity (Low, Medium, High)
• Dependent Variable (DV): Mood score (rated 1-10)

Hypotheses

• Ho: No difference in mood across exercise levels
• H1: At least one exercise level leads to a different mood
• If ANOVA is significant, a post hoc test will show which levels differ

Explained Variance

• If grouping helps explain variability, expect clear separation between groups

Unexplained Variance

• If grouping is meaningless, variability within each group will be large

Grouping

• ANOVA helps test if our grouping method is meaningful

Key Takeaways

• ANOVA is needed when comparing more than two groups
• T-tests inflate the risk of Type I errors when used multiple times
• The F-statistic compares between-group and within-group variance
• ANOVA is an omnibus test as it identifies if a difference exists, but not where
• Post hoc tests identify which groups differ
• Meaningful grouping should explain variability, not just introduce random noise
• Total Variability in Data can be separated to compare variability between groups and within groups
• If F = 1, between-group and within-group variability are equal (no real effect)
• If F > 1, there is more between-group variability, suggesting an effect
• T-tests and ANOVA work similarly because both tests examine if observed differences are due to chance

Example

• Design consists of Between-groups ANOVA where there are Different participants in each condition and is not a repeated-measures design

ANOVA Calculation

• Measures total variability in the dataset
• Square the differences to ensure positive values for variance calculation

Between-Groups Variability

• Measured due to group differences
• Formula: SSB=∑ni(Xi−X¯)2
• If groups differ more from the grand mean, SSB is larger

Within-Groups Variability

• Formula: SSW=∑(Xi−X-i)2
• Measured by random variability within each group
• If participants in the same group have similar scores, SSW is smaller

Degrees of Freedom

• Formula for Total = N-1
• Formula for Between = k-1
• Formula for Within = N-k

Mean Squares

• Formula: MS = SS/df
• Calculated using Between and Within groups

F-Statistic

• Formula: F = MSB/MSW
• If F is large, variability between groups is greater than variability within groups
• If F ≈ 1, groups are not significantly different

Determine Significance

• If p < 0.05, a significant difference is determined which rejects the null hypothesis
• At least one exercise level significantly affects mood

Key Takeaways

• ANOVA separates total variability into Between and within group variances
• Degrees of freedom adjust for sample size effects
• Calculate with the following formulas Total df = N - 1, Between df = k - 1, Within df = N - k
• The F-statistic is used to compare between-group and within-group variability where:
  • Large implies a real difference between groups and ≈ 1 means that Groups are likely not different
• If a Significant F-value is returned, it means at least one group is different, but post hoc tests are needed to find out which
• ANOVA prevents multiple comparisons error from using multiple T-tests
• ANOVA in SPSS requires that the dataset is:
  • Three conditions (Low, Medium, High)
  • Two columns for Condition (IV) and Score (DV)
  • Since this is a between-groups ANOVA, comparisons are made down the dataset

SPSS to calculate ANOVA

• Navigate using Analyze → Compare Means → One-Way ANOVA
• Adjust Options to produce:
  • Descriptive statistics, Homogeneity of variance test , Means plot, Confidence intervals
• It's best to not run post-hoc tests, in favour of these being run in Week 4

The descriptive statistics table

• Provides mean, standard deviation, and sample size for each group

Variance

• If Variance (SD) is similar across groups, a homogeneity of variance is supported that:
  • Higher exercise has been associated with better mood scores
• Levene's Test checks if variances are equal, which is a key assumption in ANOVA
• If Equal variance is not presented, you should proceed with a Welsh Test

ANOVA Output Key takeaways

• SS is a variability measurement calculated using between and within groups data
• dfBetween is calculated using k - 1 = 3 - 1 = 2.
• dfWithin : N - k = 15 - 3 = 12.

F-statistic Calculation

• Higher values equate stronger effects

Significance

• If p < 0.05, we should reject the null hypothesis
• A final message will state that At least one exercise level significantly affects mood

Assumptions in SPSS

• You can use several design-based and Statistical methods
• You can analyse these using -> Analyze --> Descriptive --> Explore--> Plots

Analysing Outputs

• SPSS fits a normal curve to each group's data however, it is best analyse the curve to check for non-normal curves
• Statistical tests can be applied using skewness, kurtosis to to check against normal distribution
• *If assumptions cannot be validated Welch Statistics using Analyze --> Compare
• It's important to first check through visual assessments to test for Homogeneity --> Data relationships, skews and outlier
• Statistical Tests can be applied to validate such relationships using Shapiro-Wilk Test: Confirms normality, Levene's Test: Checks variance equality

Purpose of One-Way ANOVA

• Key Concept: The F-Ratio between Variances (and errors

Formula

• F = Variance Between Groups \ Variance Within Groups

Relationships

• If F = 1 - > Between-group variability = Within-group variability - >No significant effect
• If F > 1 - > Between-group variability is larger - > Possible significant effect
• Note F can never be negative (unlike T-tests, which can be positive or negative)

Key Assumptions

• In ANOVA are:
  • Level of Measurement (IV->Categorical. DV -> Continuous), Random Sampling, Independence of Observance, Homogeneity and normality
• As a concept, ANOVA is an Ominous Test meaning -> ANOVA only tells us if a difference exists but not where the difference is

Steps in testing Post-Hoc

• It is a way to determine where relationships are more pronounced.
• To do so, use tests --> Tukey's test to identify which groupings present certain relationships