Abstract
Statistical power is one of the key indicators for assessing the robustness and replicability of research results. However, the standardization and completeness of calculating and reporting statistical power in event-related potential (ERP) studies still need improvement. This paper reviews and summarizes the factors influencing statistical power, methods for calculation, and application examples in ERP studies. It aims to provide researchers with a reference for calculating and reporting statistical power during the design or pre-registration stages of ERP research plans.
Introduction
The "replicability crisis" in cognitive neuroscience has brought statistical power to the forefront of methodological discussions. Many published findings in psychology and neuroscience are estimated to have a statistical power between 8% and 31%—far below the 80% gold standard. Low power not only increases the risk of false negatives (missing real effects) but also leads to inflated effect sizes in the few studies that do reach significance. Event-Related Potential (ERP) research faces unique challenges in this regard due to the hierarchical and multidimensional nature of the data.
The Complexity of ERP Data
ERP data is inherently hierarchical, with individual trials nested within subjects, who are in turn nested within experimental conditions. Traditional power analysis tools like G*Power often ignore this "trial" dimension, treating each subject as a single data point. This simplification fails to account for the variance at both the subject and trial levels, which is crucial for accurate power estimation.
As illustrated in Figure 1, the outcome of each trial contributes to the success rate for a specific subject under a specific condition. These individual rates are then aggregated to determine the population success rate. This structure requires statistical tests that can handle both continuous data (like Student's t-test) and discrete data (like the Fisher test) depending on the level of analysis.
Methods for Power Calculation
The paper categorizes three primary approaches for calculating power in ERP studies:
- Post-hoc Simulations: Using existing or pilot data to determine when an ERP component (e.g., P300 or N100) becomes "stable." Researchers use internal consistency or correlation coefficients to identify the point where adding more trials no longer significantly changes the result.
- Monte Carlo Simulations: A flexible approach where researchers generate thousands of "virtual" datasets based on parameters like expected effect size and noise levels. Power is estimated by observing how often the null hypothesis is correctly rejected across these simulations.
- Power Contour Plots: A visual method mapping the trade-off between the number of subjects ($N$) and the number of trials ($k$), allowing researchers to find the most cost-effective way to achieve 80% power.
Key Factors Influencing Power: The N vs. k Trade-off
A critical decision in ERP study design is balancing the number of participants ($N$) and the number of trials ($k$). While increasing $k$ improves the signal-to-noise ratio (SNR) by the square root of $k$, increasing $N$ often has a more dramatic impact on statistical power, especially in between-subject designs.
Figure 2 highlights how the impact of these factors varies by design. In within-subject designs, increasing the number of trials has a very strong positive impact (++) on power. Conversely, in between-subject designs, increasing the number of subjects is the more dominant factor (++) for boosting power. However, researchers must be wary of the "ceiling effect," where adding trials beyond the point of component stability (e.g., ~40 trials for the ERN) yields diminishing returns and may increase noise due to participant fatigue.
Conclusion and Recommendations
To address the replicability crisis, ERP researchers must move toward more sophisticated power analyses that account for the hierarchical nature of their data. The authors recommend using pilot data or publicly available datasets to inform simulations and utilizing open-source tools like the ERP Power Calculator. Finally, including these power analyses in pre-registered reports is essential for ensuring that studies are designed with sufficient "muscle" to produce robust, replicable findings.
Summary updated