Within-Subjects ANOVA

PSY 348: Lecture 14

Dave Brocker

Farmingdale State College

When to Use Repeated Measures ANOVA

Use Repeated Measures ANOVA when:

  • You have the same participants measured more than once (e.g., pre-test/post-test).

  • You’re interested in changes over time or under different conditions.

    • Testing whether students’ enjoyment of statistics improves after receiving lollipops for 2 weeks.

Between vs. Within Subjects ANOVA

Between-subjects ANOVA

  • Compares means between groups (Group A vs. B vs. C).

  • Error comes from individual differences between people.

Within-subjects (repeated measures) ANOVA

  • Compares means within the same participants over time or conditions.

  • Controls for individual differences.

Pizza and Productivity Study

A Repeated-Measures Design

Design

  • Objective: test whether productivity changes over time.

  • IV: time

    1. Before lunch
    2. 10 minutes after pizza
    3. 1 hour after pizza

  • The same participants are measured at each timepoint.

  • DV: number of emails written in 10 minutes.

Visualizing the Data

Why Repeated-Measures ANOVA?

  • Can’t use multiple paired-samples t-tests.

    • Alpha inflation increases the Type I error rate.

  • Repeated-Measures ANOVA allows us to:

    • Compare within-subject changes across time.

    • Control for individual variability.

    • Test for an overall time effect.

Within-Subjects ANOVA

Where the variance comes from

  • Between-subjects ANOVA splits variance into between-group and error (within-group) parts.

  • Within-subjects ANOVA has a third source, because each person contributes several scores:

Time
How the means vary across the time points (the effect we care about).
Subjects
How people differ from each other overall.
Error
What is left over once time and subjects are accounted for.

Within-Subjects ANOVA

Partitioning the variability

A Total Variability B Time Variability A->B C Within-Groups Variability A->C D Subject Variability C->D E Error Variability C->E

Within-Subjects ANOVA

The key idea

Removing the person

\(SS_{W} = SS_{Subjects} + SS_{Error}\)

\(SS_{Error} = SS_{W} - SS_{Subjects}\)

  • In between-subjects ANOVA, differences between people sit inside the error term.

  • In within-subjects ANOVA we subtract them out, so the error is smaller and the test is more powerful.

Calculating Repeated-Measures ANOVA

The F ratio

\(F = \dfrac{MS_{Time}}{MS_{Error}}\)

\(MS_{Time} = \dfrac{SS_{Time}}{df_{Time}} = \dfrac{SS_{Time}}{k - 1}\)

\(MS_{Error} = \dfrac{SS_{W} - SS_{Subjects}}{df_{Error}} = \dfrac{SS_{W} - SS_{Subjects}}{(n-1)(k-1)}\)

  • \(k\) is the number of time points (or conditions) and \(n\) is the number of participants.

  • Each SS is a sum of squared distances from a mean (\(\bar{x}_t\) = a time mean, \(\bar{x}_p\) = a person’s mean, \(\bar{x}_G\) = the grand mean): \(SS_{Time} = \sum n(\bar{x}_t - \bar{x}_G)^2 \quad SS_{Subjects} = \sum k(\bar{x}_p - \bar{x}_G)^2 \quad SS_{W} = \sum (x - \bar{x}_t)^2\)

Degrees of Freedom

Rules

  • Time: \(df_{Time} = k - 1\)

  • Subjects: \(df_{Subjects} = n - 1\)

  • Error: \(df_{Error} = (n - 1) \times (k - 1)\)

Degrees of Freedom

Example

20 students rate their stress at three time points during finals week: three days before the exam, the night before, and after the exam.

What are the df’s for a one-way repeated-measures ANOVA on Time?

Reveal answer

  • \(df_{Time} = 3 - 1 = 2\)

  • \(df_{Subjects} = 20 - 1 = 19\)

  • \(df_{Error} = 19 \times 2 = 38\)

Pizza Study Results

The ANOVA table

Source df SS MS F p
Time (condition) 2 105.58 52.79 69.19 <.001
Subjects 29 175.71 6.06 – –
Error 58 44.25 0.76 – –

\(F(2, 58) = 69.19\), \(p\) < .001: productivity changes significantly across the three time points.

Pizza Study Results

In R

repeated_measures.R

library(afex)

# id = who; within = the repeated factor
fit <- aov_ez(id = "id", dv = "y", data = pizza,
              within = "cond", return = "afex_aov")

# the corrected ANOVA table (Greenhouse-Geisser if needed)
anova(fit)
Anova Table (Type 3 tests)

Response: y
     num Df den Df     MSE      F     ges    Pr(>F)    
cond 1.9264 55.866 0.79209 69.189 0.32431 1.403e-15 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
  • summary(fit) also prints Mauchly’s test and the other corrections.

Pizza Study Results

Why within-subjects helps

69.19

Repeated-measures F (removes person differences)

20.88

Between-subjects F (same data, ignoring the person)

  • The same scores give a much larger F once we account for the fact that each person was measured three times.

Repeated-Measures ANOVA

Assumptions

Assumptions

  • Normality of the differences.

  • Sphericity: the variances of all pairwise differences are about equal.

    • Test: Mauchly’s Test.

  • If sphericity is violated, use the Greenhouse-Geisser correction.

    • Always check sphericity when the within-subject factor has more than two levels.

Repeated-Measures ANOVA

Interpretation

  • \(F(2, 58) = 69.19\), \(p\) < .001

    • Main effect of time on productivity.

    • Productivity changes significantly across the timepoints.

      • What would be your next step?

Repeated-Measures ANOVA

Effect size

Partial eta squared

\(\eta_p^2 = \dfrac{SS_{Time}}{SS_{Time} + SS_{Error}} = \dfrac{105.58}{105.58 + 44.25} =0.70\)

  • Benchmarks are the same as other ANOVAs: about .01 small, .06 medium, .14 large.

Follow-Up

Post-hoc comparisons

  • A significant F says that some time points differ, not which.

  • Run paired t-tests with a Bonferroni correction:

posthoc.R

pairwise.t.test(pizza$y, pizza$cond,
                paired = TRUE, p.adjust.method = "bonferroni")

    Pairwise comparisons using paired t tests 

data:  pizza$y and pizza$cond 

             Before lunch 10 min after
10 min after 1.5e-08      -           
1 hour after 0.0086       1.4e-12     

P value adjustment method: bonferroni

Follow-Up

Reading the comparisons

Comparison Adjusted p
Before lunch vs. 10 min after <.001
Before lunch vs. 1 hour after 0.009
10 min after vs. 1 hour after <.001
  • The adjusted p-values already account for running three comparisons.

Within-Subjects ANOVA

Reporting F

If asked to report findings in terms of the Null Hypothesis (H0), you should report findings as:

  • Reject \(H_0\), or

  • Fail to Reject \(H_0\)

  • If asked to report findings in general or for publication, you need to report \(F(df_{Time}, df_{Error}) = F, p\), an effect size, and the means and standard deviations at each time point.

Within-Subjects ANOVA

Reporting the pizza study

Productivity differed significantly across the three time points, F(2, 58) = 69.19, p < .001, \(\eta_p^2\) = 0.70. Productivity was highest 10 minutes after pizza (M = 6.93, SD = 1.61) and lowest an hour after pizza (M = 4.33, SD = 1.60); before lunch it was M = 5.14 (SD = 1.56).

Repeated-Measures ANOVA

Example: an engagement intervention

Prof. Brocker evaluates a new engagement intervention. 100 students rate their enjoyment of statistics (1 to 10) at the start of the semester, after week 1 of a statistics warm-up activity, and after week 2.

Find the degrees of freedom

  • \(df_{Time}\) = ?

  • \(df_{Subjects}\) = ?

  • \(df_{Error}\) = ?

  • \(n\) = ?

Repeated-Measures ANOVA

Engagement intervention: answers

Degrees of freedom

  • \(df_{Time} = k - 1 = 3 - 1 = 2\)

  • \(df_{Subjects} = n - 1 = 100 - 1 = 99\)

  • \(df_{Error} = (n - 1)(k - 1) = 99 \times 2 = 198\)

  • \(n = 100\)

Repeated-Measures ANOVA

Engagement intervention: results

Source df SS MS F p
Time (condition) 2 111.72 55.86 343.60 <.001
Subjects 99 769.76 7.78 – –
Error 198 32.19 0.16 – –

F(2, 198) = 343.60, p < .001

Error bars: mean \(\pm\) SE, where \(SE = \dfrac{s}{\sqrt{n}}\) and \(s\) is the SD of the scores.

Repeated-Measures ANOVA

Example: active-learning practices

Professor Brocker tests whether structured active-learning practices improve 80 students’ attitudes toward statistics. He collects a 1-10 enjoyment rating at the beginning of the semester, mid-semester (after four modules), and the end (after eight modules).

Find the degrees of freedom

  1. \(df_{Time}\) = ?
  2. \(df_{Participants}\) = ?
  3. \(df_{Error}\) = ?

Repeated-Measures ANOVA

Active-learning practices: answers

Degrees of freedom

  1. \(df_{Time} = k - 1 = 3 - 1 = 2\)
  2. \(df_{Participants} = n - 1 = 80 - 1 = 79\)
  3. \(df_{Error} = (n - 1)(k - 1) = 79 \times 2 = 158\)

Repeated-Measures ANOVA

Active-learning practices: results

Source df SS MS F p
Time (condition) 2 116.75 58.38 281.96 <.001
Subjects 79 552.90 7.00 – –
Error 158 32.71 0.21 – –

F(2, 158) = 281.96, p < .001

Error bars: mean \(\pm\) SE, where \(SE = \dfrac{s}{\sqrt{n}}\) and \(s\) is the SD of the scores.

Interpreting F

What F means

\(MS_{Time}\) is made up of \(MS_{Error}\) + the theoretical change over time:

  • \(MS_{Time} = \text{change over time} + MS_{Error}\)

  • If there is no change: \(MS_{Time} = 0 + MS_{Error}\), so \(F = \dfrac{MS_{Error}}{MS_{Error}} = 1\).

  • If there is a change: \(F = 1 + \dfrac{\text{change}}{MS_{Error}}\), which is greater than 1.

  • When \(F \leq 1\), it’s not likely to be significant.

Hypothesis Testing: ANOVA

Decision rule

  • \(F \leq 1\): fail to reject the null hypothesis.

  • \(F > 1\): compare to the critical value (or use the p-value). Reject \(H_0\) only if \(p < \alpha\).

Reporting F

Active-learning example

Attitudes toward statistics differed significantly across the semester, F(2, 158) = 281.96, p < .001. Attitudes were highest at the end of the semester (M = 6.27, SD = 1.69) and lowest at the beginning (M = 4.56, SD = 1.42). Mid-semester attitudes were M = 5.45 (SD = 1.59).

Data Manipulation

Wide and long format

  • When collected, data can be in either wide or long format.

  • Below are the same five participants in both formats.

Wide data
Each row is one participant
Participant Immediate After24Hours After1Week
1 11.4 4.6 4.5
2 6.2 7.1 8.1
3 7.0 5.8 7.4
4 7.4 6.2 4.5
5 6.5 9.3 6.8

Data Manipulation

Wide and long format

Long data
Each row is one participant at one time
Participant Time Memory
1 Immediate 11.4
1 After24Hours 4.6
1 After1Week 4.5
2 Immediate 6.2
2 After24Hours 7.1
2 After1Week 8.1
3 Immediate 7.0
3 After24Hours 5.8
3 After1Week 7.4
4 Immediate 7.4
4 After24Hours 6.2
4 After1Week 4.5
5 Immediate 6.5
5 After24Hours 9.3
5 After1Week 6.8

pivot.R

library(tidyr)

# wide -> long: one row per person per time
long <- wide |>
  pivot_longer(cols = !Participant,
               names_to = "Time",
               values_to = "Memory")
  • aov() and aov_ez() need the long format.

Practice

1. Memory recall

30 participants study a word list and recall it immediately, after 24 hours, and after 1 week. Does recall change over time?

Source df SS MS F p
Time (condition) 2 213.74 106.87 102.22 <.001
Subjects 29 155.67 5.37 – –
Error 58 60.64 1.05 – –
Reveal answer

F(2, 58) = 102.22, p < .001: recall decreased significantly over time.

Practice

2. Clinical trial

30 patients each receive a low, medium, and high dose of a medication (in separate sessions) and report symptom improvement. Does dose matter?

Source df SS MS F p
Time (condition) 2 89.14 44.57 60.95 <.001
Subjects 29 178.95 6.17 – –
Error 58 42.41 0.73 – –
Reveal answer

F(2, 58) = 60.95, p < .001: improvement increased significantly with dose.

Practice

3. Behavioral therapy progress

30 participants rate their anxiety at baseline, mid-treatment, and post-treatment.

Source df SS MS F p
Time (condition) 2 434.22 217.11 101.90 <.001
Subjects 29 268.05 9.24 – –
Error 58 123.58 2.13 – –
Reveal answer

F(2, 58) = 101.90, p < .001: anxiety decreased.

Practice

4. Consumer product ratings

30 consumers each rate three products (A, B, and C).

Source df SS MS F p
Time (condition) 2 73.17 36.58 39.80 <.001
Subjects 29 215.88 7.44 – –
Error 58 53.31 0.92 – –
Reveal answer

F(2, 58) = 39.80, p < .001: the ratings differed by product. A post-hoc test would show which products differ.

Practice

5. Physiological stress

30 participants’ heart rate is measured at rest, during a task, and during recovery.

Source df SS MS F p
Time (condition) 2 7,027.50 3,513.75 215.83 <.001
Subjects 29 2,266.64 78.16 – –
Error 58 944.26 16.28 – –
Reveal answer

F(2, 58) = 215.83, p < .001: heart rate differed significantly across the phases.

Summary: Within-Subjects ANOVA

  • Compares three or more related means (the same people measured repeatedly).

  • Splits the variability into time, subjects, and error, and removes the subject differences from the error term.

  • \(F = MS_{Time} / MS_{Error}\) with \(df = (k - 1,\ (n - 1)(k - 1))\).

  • Check sphericity (Mauchly’s Test) and use the Greenhouse-Geisser correction if it is violated.

  • Follow a significant \(F\) with post-hoc paired comparisons (Bonferroni).

  • Report \(F(df_{Time}, df_{Error})\), \(p\), an effect size, and the means and standard deviations.