Abstract
Hierarchically-organized data arise naturally in many psychology and neuroscience studies. As the standard assumption of independent and identically distributed samples does not hold for such data, two important problems are to accurately estimate group-level effect sizes, and to obtain powerful statistical tests against group-level null hypotheses. A common approach is to summarize subject-level data by a single quantity per subject, which is often the mean or the difference between class means, and treat these as samples in a group-level t-test. This "naive" approach is, however, suboptimal in terms of statistical power, as it ignores information about the intra-subject variance. To address this issue, we review several approaches to deal with nested data, with a focus on methods that are easy to implement. With what we call the sufficient-summary-statistic approach, we highlight a computationally efficient technique that can improve statistical power by taking into account within-subject variances, and we provide step-by-step instructions on how to apply this approach to a number of frequently-used measures of effect size. The properties of the reviewed approaches and the potential benefits over a group-level t-test are quantitatively assessed on simulated data and demonstrated on EEG data from a simulated-driving experiment.
TL;DR
When analyzing nested data (like multiple trials per subject), the common practice of simply averaging trials and running a group-level t-test wastes significant statistical power. This paper advocates for the "sufficient-summary-statistic" approach: by carrying both the mean and the variance of each subject into the group analysis, researchers can achieve much more sensitive results without the computational complexity of full linear mixed models.
The Challenge of Nested Data
In neuroscience and psychology, data is almost always hierarchical: we record many trials (Level 1) from several subjects (Level 2). Standard statistical tests like the t-test assume that every data point is independent and identically distributed (i.i.d.). Nested data violates this because trials within one subject are more similar to each other than to trials from another subject.
Researchers usually handle this in one of two ways:
- Data Pooling: Throwing all trials from all subjects into one giant test. This is dangerous; it can lead to massive false-positive rates or completely miss effects due to between-subject variability.
- The "Naive" Summary Approach: Calculating a mean for each subject and running a t-test on those means. While statistically valid, this is "suboptimal" because it treats a subject with 100 clean, consistent trials the same as a subject with 10 noisy trials. It ignores the "certainty" (intra-subject variance) of each subject's data.
The Sufficient-Summary-Statistic Approach
The paper focuses on the Sufficient-Summary-Statistic Approach. Instead of passing just the mean ($\hat{\theta}_s$) to the group level, you pass both the mean and its estimated variance ($\hat{\sigma}^2_s$).
Key components include:
- Inverse-Variance Weighting: Rather than giving every subject equal weight, the group effect is calculated by weighting each subject by the reciprocal of their variance. Subjects with more reliable data (lower variance) have a larger say in the group result.
- Random-Effects (RE) Modeling: The authors use the DerSimonian and Laird approach to estimate between-subject variance. This ensures the model accounts for the fact that the "true" effect might naturally differ from person to person.
- Broad Applicability: They provide step-by-step instructions for applying this to various effect sizes, including differences between means (ERP analysis), Area Under the ROC Curve (AUC) for decoding, and Pearson correlations (using Fisher z-transformation).
Key Insights: Why Pooling Fails
- The "Aha" Moment: You don't need to fit a computationally expensive Linear Mixed Model (LMM) to get high statistical power. If you can estimate the variance of your subject-level effect, you have "sufficient statistics" to perform nearly optimal group inference.
- Pooling is a Trap: The paper demonstrates that pooling data can create "spurious correlations." For example, if subjects have different baseline activity levels, pooling can show a strong correlation between two variables at the group level even if zero correlation exists within any individual subject.
- Fixed vs. Random Effects: A "Fixed-Effect" model assumes every subject has the exact same underlying effect. If this assumption is wrong, the model becomes overconfident, leading to a high rate of false positives (up to 26% in their simulations).
- The Weighting Caveat: If a subject's effect size is correlated with their variance, inverse-variance weighting can introduce bias. In these specific cases, equal weighting is actually safer.
Key Results
- Superior Power: In simulations, the inverse-variance weighted approach consistently showed higher power for true effects compared to the naive t-test, especially when trial counts or noise levels varied between subjects.
- Robustness: The naive t-test maintained a 5% false-positive rate but lacked power. The proposed RE sufficient-summary approach matched the 5% false-positive rate while significantly boosting power.
- EEG Application: When applied to driving data, the sufficient-summary approach provided a much clearer statistical separation between conditions than the naive approach, correctly mapping the temporal order of brain and muscle activity.
Limitations and Future Work
- Small Sample Sizes: With very few subjects (e.g., $S=5$), estimating between-subject variance becomes unstable, which can slightly inflate false-positive rates.
- Correlation Bias: The method assumes effect size and variance are independent. Future work could explore more robust weighting schemes for cases where this doesn't hold.
- Non-Linearity: The approach relies on the distribution of summary statistics being approximately normal. For extremely non-normal trial distributions, further refinements might be needed.
Summary updated