Analysis of a large fMRI cohort: Statistical and methodological issues for group analyses

Bertrand Thirion, Philippe Pinel, Sébastien Mériaux, Alexis Roche, Stanislas Dehaene, Jean-Baptiste Poline

TL;DR

This paper investigates the reliability of fMRI group analysis, a crucial step for understanding brain function. By analyzing a large dataset, the authors reveal that inter-subject variability significantly impacts results and propose that optimal statistical thresholds and sample sizes are key to achieving reliable findings.

What Problem Does This Paper Address?

Functional Magnetic Resonance Imaging (fMRI) is a powerful tool for studying brain activity. However, individual brain scans are highly variable, making it challenging to draw general conclusions about brain function from small groups of subjects. Researchers typically use statistical methods to combine data from multiple individuals (group analysis) to identify consistent patterns of brain activation. A major challenge in this field is ensuring the reliability of these group findings – meaning that the observed patterns are consistent and not just due to chance or specific characteristics of the chosen subjects. This paper addresses the critical question of how to achieve reliable fMRI group analyses by exploring the impact of various methodological choices, including statistical thresholds, sample size, and different analysis techniques, on the reproducibility of results.

Methods

The authors employ a bootstrap approach to assess the reliability of fMRI group analyses. They repeatedly divide their large dataset into smaller, independent subgroups of varying sizes. For each subgroup, they perform standard fMRI analysis and derive statistical maps. To quantify reliability, they introduce two main measures:

  1. Voxel-level reliability: This measure models the "active/inactive" status of each voxel as a mixture of binomial distributions. It quantifies how consistently a voxel is identified as active across different subgroups.
  2. Cluster-level reliability: This measure focuses on the spatial consistency of supra-threshold clusters. It calculates the average distance between the centers of mass of these clusters across different subgroups, with a smaller distance indicating higher reliability.

They also compare the performance of several statistical methods, including Random Effects (RFX), Mixed Effects (MFX), and non-parametric approaches like Wilcoxon's signed rank test, under different conditions.

Datasets

The study utilizes a large dataset collected from an event-related fMRI "localizer" experiment. This dataset comprises data from 81 right-handed subjects. The specific experimental paradigm involved various tasks, including passive viewing, button presses, and auditory/visual instructions.

Key Insights and Ideas

  • Inter-subject variability is a major hurdle: The study strongly emphasizes that variability between individuals is a primary driver of low sensitivity and reliability in fMRI group studies. This variability stems from factors like spatial misregistration and differences in signal magnitude.
  • Optimal thresholds exist and are lower than often used: Contrary to common practice, the authors find that a relatively lenient statistical threshold (around p < 0.001, uncorrected) can lead to more reliable results, suggesting a better balance between detecting true positives and avoiding false positives.
  • Sample size matters significantly: The research clearly demonstrates that reliability increases with the number of subjects. A plateau in reliability is observed for group sizes of 20 or more, suggesting that many current fMRI studies may lack sufficient statistical power for reliable findings.
  • Non-parametric methods are preferred: The study advocates for non-parametric statistical assessments over parametric ones, as they rely on fewer assumptions about the data distribution, leading to more robust and reliable results.
  • Cluster-level analysis is more reliable than voxel-level: Thresholding at the cluster level, rather than individual voxels, generally improves reliability, likely by accounting for spatial smoothness and reducing the impact of random noise at individual voxels.

Key Results

  • Reliability plateau at ~20 subjects: The study found that reliability measures plateaued when the group size reached approximately 20 subjects.
  • Optimal threshold around θ* ~ 2.7-3: The optimal statistical threshold for binarizing maps, balancing reproducibility and sensitivity, was found to be around 2.7 to 3.
  • Method comparison: Among the tested methods, the pseudo-MFX (ΨFX) technique showed the highest reliability, though its interpretation is less straightforward. Mixed-effects (MFX) and cluster-based methods also demonstrated good reliability.
  • Contrast-dependent reliability: The reliability of findings varied depending on the cognitive contrast being studied, with auditory tasks showing higher reproducibility than motor or computation tasks.

Limitations and Future Work

The authors acknowledge that their study is not exhaustive and that other techniques could have been explored. They also note that the ΨFX statistic, while highly reliable, lacks a clear interpretation and requires voxel-wise thresholding.

Future work suggested by the authors includes:

  • Relating inter-subject variability to individual behavioral and psychological characteristics.
  • Investigating the link between spatial functional variability and anatomical variability, potentially using diffusion imaging.
  • Exploring intermediate levels of description between individual subjects and the group, such as identifying outlier subjects or subgroups.
  • Developing more systematic methods for comparing pre-processing techniques and analysis frameworks.
  • Further investigating the question of optimized thresholds for neuroimaging studies.

Summary updated

Abstract

The aim of group fMRI studies is to relate contrasts of tasks or stimuli to regional brain activity increases. These studies typically involve 10 to 16 subjects. The average regional activity statistical significance is assessed using the subject to subject variability of the effect (random effects analyses). Because of the relatively small number of subjects included, the sensitivity and reliability of these analyses is questionable and hard to investigate. In this work, we use a very large number of subject (more than 80) to investigate this issue. We take advantage of this large cohort to study the statistical properties of the inter-subject activity and focus on the notion of reproducibility by bootstrapping. We asked simple but important methodological questions: Is there, from the point of view of reliability, an optimal statistical threshold for activity maps? How many subjects should be included in group studies? What method should be preferred for inference? Our results suggest that i ) optimal thresholds can indeed be found, and are rather lower than usual corrected for multiple comparison thresholds, ii ) 20 subjects or more should be included in functional neuroimaging studies in order to have sufficient reliability, iii ) non-parametric significance assessment should be preferred to parametric methods, iv ) cluster-level thresholding is more reliable than voxel-based thresholding, and v ) mixed effects tests are much more reliable than random effects tests. Moreover, our study shows that inter-subject variability plays a prominent role in the relatively low sensitivity and reliability of group studies. Previous article in issue Next article in issue Recommended articles Citing articles (355) View full text Copyright © 2006 Elsevier Inc. All rights reserved.
BibTeX
@article{thirion2007analysis,
  author = {Bertrand Thirion and Philippe Pinel and Sébastien Mériaux and Alexis Roche and Stanislas Dehaene and Jean-Baptiste Poline},
  title = {{Analysis of a large fMRI cohort: Statistical and methodological issues for group analyses}},
  journal = {NeuroImage},
  year = {2007},
  doi = {10.1016/j.neuroimage.2006.11.054},
}
Figure