Permutation methods for factor analysis and PCA

Edgar Dobriban

Abstract: Researchers often have datasets measuring features $x_{ij}$ of samples, such as test scores of students. In factor analysis and PCA, these features are thought to be influenced by unobserved factors, such as skills. Can we determine how many components affect the data? This is an important problem, because decisions made here have a large impact on all downstream data analysis. Consequently, many approaches have been developed. Parallel Analysis is a popular permutation method: it randomly scrambles each feature of the data. It selects components if their singular values are larger than those of the permuted data. Despite widespread use, as well as empirical evidence for its accuracy, it currently has no theoretical justification. In this paper, we show that parallel analysis (or permutation methods) consistently select the large components in certain high-dimensional factor models. However, when the signals are too large, the smaller components are not selected. The intuition is that permutations keep the noise invariant, while “destroying” the low-rank signal. This provides justification for permutation methods. Our work also uncovers drawbacks of permutation methods, and paves the way to improvements.

Introduction

In fields ranging from genomics to psychology, researchers use Principal Component Analysis (PCA) and Factor Analysis (FA) to find hidden patterns in massive datasets. The most critical—and often most subjective—step is deciding the "rank" of the data: how many underlying factors (like "intelligence" in a battery of test scores) are actually driving the variation?

For years, the gold standard has been Parallel Analysis (PA). The idea is simple: take your data, randomly scramble the entries in each column to destroy any structure, and see what the "noise" looks like. If a factor in your original data is stronger than the factors found in this scrambled "null" data, you keep it. While PA is famous for being accurate in practice, it was essentially a "black box" with no mathematical proof explaining why it works or when it might fail.

Methodology and Theoretical Framework

This research analyzes PA through the lens of high-dimensional asymptotics, examining what happens as both the number of samples ($n$) and the number of features ($p$) grow toward infinity. The paper introduces a Consistency Lemma based on two core principles:

  1. Noise Invariance: Shuffling the data shouldn't fundamentally change the distribution of the random noise already present.
  2. Signal Destruction: Shuffling a "smooth," low-rank signal (the real factors) should transform it into a "rough," high-rank matrix with a much smaller impact on the data's overall magnitude (operator norm).
Visualization of signal destruction: a smooth low-rank signal (left) becomes high-rank noise (right) after permutation.

By proving these two conditions hold under high-dimensional settings, the author provides the first formal justification for using permutation methods to identify significant components. The work primarily relies on rigorous mathematical proofs, validated by synthetic numerical simulations involving one-factor and two-factor models with varying signal-to-noise ratios and sparsity.

Key Insights

The Blessing of Dimensionality

Counter-intuitively, PA works better when the number of features ($p$) is large relative to the sample size ($n$). In low-dimensional settings, shuffling can lead to overestimating noise, making the method too conservative. Furthermore, for PA to work, a factor must be "delocalized"—meaning it influences many variables. If a factor only affects one or two variables, shuffling those columns doesn't "destroy" the signal enough to distinguish it from noise.

The "Shadowing" Effect

One of the paper's most striking findings is the shadowing effect. If you have one extremely strong factor and one moderately weak factor, the strong factor—once shuffled—creates so much "fake noise" that it can drown out (or "shadow") the weaker factor, causing PA to miss it entirely.

Comparison of singular values for original data (X, dashed) and permuted data (X_pi, solid). The signal in X is clearly visible above the noise floor of X_pi at low k.

Results and Limitations

The paper proves that PA consistently selects all "perceptible" factors (those significantly above the noise floor) as long as the data is high-dimensional and the factors are sufficiently spread out. Simulations showed PA is remarkably robust; it could correctly identify a factor even if it only loaded on 2% of the variables.

However, the Shadowing Effect remains a significant limitation. Because PA transforms real signals into "null" noise, very large signals bias the threshold upward. For example, in a two-factor model, if the primary signal is more than ~35 times the noise level, it can prevent the detection of a secondary signal that is 6 times the noise level. The author suggests that a "deflation" approach—identifying and removing the strongest factors one by one before re-running the permutation—could solve this, a direction for future research.

Summary updated

Abstract

Researchers often have datasets measuring features $x_{ij}$ of samples, such as test scores of students. In factor analysis and PCA, these features are thought to be influenced by unobserved factors, such as skills. Can we determine how many components affect the data? This is an important problem, because decisions made here have a large impact on all downstream data analysis. Consequently, many approaches have been developed. Parallel Analysis is a popular permutation method: it randomly scrambles each feature of the data. It selects components if their singular values are larger than those of the permuted data. Despite widespread use, as well as empirical evidence for its accuracy, it currently has no theoretical justification. In this paper, we show that parallel analysis (or permutation methods) consistently select the large components in certain high-dimensional factor models. However, when the signals are too large, the smaller components are not selected. The intuition is that permutations keep the noise invariant, while “destroying” the low-rank signal. This provides justification for permutation methods. Our work also uncovers drawbacks of permutation methods, and paves the way to improvements.
BibTeX
@article{dobriban2020permutation,
  author = {Edgar Dobriban},
  title = {{Permutation methods for factor analysis and PCA}},
  journal = {The Annals of Statistics},
  year = {2020},
  doi = {10.1214/19-aos1907},
}
Figure