Principal component analysis (PCA) finds the directions along which data vary most. When there are about as many measured features as samples, or more, the leading direction it returns can be almost unrelated to the true one. Theorists repair this with sparse PCA, which assumes the true direction involves only a few features. That literature now offers sharp guarantees: minimax error rates, phase-transition thresholds, and polynomial-time algorithms that attain them. Nearly all of these guarantees are checked only on synthetic Gaussian data with spherical background noise. The reason is simple. On real data the true direction is unknown, so the quantity the theory bounds, the angle between the estimated and true directions, cannot be measured. As a result we do not know whether the theory's rankings of methods, its predicted sample-size thresholds, or its failure modes hold on gene expression, text, or stock returns.
This note proposes a fix. We take real high-dimensional data as the background noise and add ("plant") a sparse direction whose identity we control. The planted direction is the ground truth, while the noise keeps its real correlations, heavy tails, and leftover biological or economic structure. A two-by-two set of controls then separates which idealized assumption, spherical noise or Gaussianity, causes any gap between theory and practice. A ground-truth-free check based on agreement between independent data halves is calibrated on the planted data, so it can be trusted on unplanted real data and used to choose the sparsity level.
Core Claim
Real-background planting is a measurement instrument for the empirical validation gap in high-dimensional PCA. It yields three kinds of evidence that synthetic studies and downstream-accuracy studies cannot.
The first is empirical phase-transition curves on real noise. These show whether the predicted scaling of success with $n/(s\log p)$ holds. Here $n$ is the sample count, $s$ the number of nonzero features in the true direction, and $p$ the number of features.
The second is a decomposition of each method's failures into the part caused by correlated noise and the part caused by non-Gaussian marginals.
The third is a calibrated stability statistic for selecting sparsity without an oracle.
The concrete prediction, which can be falsified, concerns how the curves behave once sample size is rescaled. If the theory transfers, success-probability curves for a given method should collapse onto one curve across backgrounds. The rescaling is by the effective noise level $\sigma^2$ from Minimax Rates of Estimation for Sparse PCA in High Dimensions, which Vu and Lei show is governed by the eigengap and which we compute from a large-sample reference covariance. Where the curves fail to collapse, we can name the assumption responsible.
Why this, and not the obvious study
The obvious intersection project is to run provable sparse PCA estimators as feature extractors on the TCGA Meta-Dataset Clinical Benchmark and report classification accuracy. That benchmark has 174 tasks, 20,530 genes, and 50 training samples per task. It is a weak test of the theory. Downstream accuracy mixes estimation quality with how well the label happens to align with high-variance directions. The reported margins are also small relative to the variation across tasks: logistic regression reaches 68.78% against 66.18% for always predicting the majority class, with standard deviations near 16%.
The closest existing empirical work also stops short of measuring recovery. Combinatorial Sparse PCA Beyond the Spiked Identity Model motivates our design. It proves that diagonal thresholding can fail completely under correlated noise, and it evaluates its restarted truncated power method on NYTimes text. However, it scores the text results only by topic coherence and explained variance, with no ground truth. Sparse PCA: Convex Relaxations, Algorithms and Applications and Sparse PCA with False Discovery Rate Controlled Variable Selection validate on S&P 500 returns qualitatively, by recovering known industry sectors.
Tighten after Relax, Fantope Projection and Selection, Oja's Algorithm for Streaming Sparse PCA, and A Kernel Random Matrix-Based Approach for Sparse PCA test only on synthetic data. Our design differs from all of them in one respect: the truth is known and the noise is real.
Research Program
Backgrounds. We use three real backgrounds named in the file:
- TCGA RNA-seq expression, pooling roughly 11,000 tumors so that a large reference set exists. We stratify by cancer type so the planted signal does not simply compete with tissue-of-origin.
- The NYTimes bag-of-words matrix used by Kumar et al., with 10,000 documents and 20,000 words.
- S&P 500 daily returns from the 2005–2010 window, with 472 stocks.
For each background we hold out a large reference pool and compute its covariance $\Sigma_{\text{ref}}$. Experimental samples of size $n$ are drawn from the remainder.
Planting. We form
$$x_i = b_i + \sqrt{\beta}\, z_i\, v,$$
where:
- $b_i$ is a real background row (centered and scaled per feature);
- $v$ is a unit vector with $s$ nonzero entries;
- $z_i$ is a scalar latent factor, Gaussian in the main runs and Student-t in a robustness arm;
- $\beta$ sets the signal strength.
The planted signal must compete with the background's own top directions. We therefore calibrate $\beta$ relative to $v^\top\Sigma_{\text{ref}}v$ and to the background's leading eigenvalues, and we report the resulting eigengap.
The support of $v$ is placed in one of three ways: uniformly at random; on the highest-variance features, which directly tests the "noise trap" of Machkour et al.; or on features strongly correlated with each other, which tests the hard constructions of Kumar et al.
The two-by-two control. For each background we build four noise sources that keep or remove the real correlation structure and the real marginal distributions:
- Gaussian, real correlations. Gaussian noise with covariance exactly $\Sigma_{\text{ref}}$, generated cheaply as $X_{\text{ref}}^\top g/\sqrt{N}$ with $g$ standard normal and $N$ the size of the reference pool.
- Real marginals, no correlations. The real background with each column permuted independently.
- Fully real. The real background itself.
- Fully idealized. Spherical Gaussian noise rescaled to match the per-feature variances.
Any change in a method's phase-transition location across these four cells attributes the empirical gap to correlation, non-Gaussianity, or their interaction.
A truth-free check, calibrated. We split the data into two independent halves and estimate the sparse direction on each. Under a spiked model the two estimates are approximately independent given the truth. Their overlap therefore estimates the squared alignment with the truth:
$$|\langle \hat v^{(1)}, \hat v^{(2)}\rangle| \approx \langle \hat v, v\rangle^2 .$$
We will measure, on planted data, how closely this relation holds in each control cell. Where it holds, the statistic serves two purposes. It estimates recovery error on unplanted real components. It also gives a rule for choosing $s$ or the penalty strength: pick the most stable sparsity level.
We compare this rule with cross-validation, which Inconsistency of Cross-Validation for Structure Learning in Gaussian Graphical Models shows overselects in related high-dimensional structure learning. We also compare it with the false-discovery-rate selector of Machkour et al.
Baselines
Ordinary PCA. The do-nothing reference is ordinary dense PCA on the same samples. It shows where the classical inconsistency appears on real noise.
Diagonal thresholding. This keeps the features with the largest sample variances and then runs PCA on them. It is the fast heuristic whose sample requirement High-dimensional analysis of semidefinite relaxations for sparse principal components shows scales as $k^2\log p$, against $k\log p$ for the semidefinite relaxation. Kumar et al. predict it fails outright under correlated noise. Our real-correlation cells test that prediction directly.
Restarted truncated power method. This is the strongest competitor under general covariance. It runs truncated power iterations from every coordinate vector and keeps the highest-variance result (Combinatorial Sparse PCA Beyond the Spiked Identity Model). It is the only method in the file with both general-covariance guarantees and demonstrated scaling to $d=20{,}000$.
Tighten after Relax. This method first solves a convex relaxation by ADMM and stops early, then refines the result with Sparse Orthogonal Iteration Pursuit. Its paper reports the strongest number in the file: projection-matrix Frobenius error 0.32 against 1.15 for TPower in a small-eigengap synthetic setting. We will reproduce that synthetic setting first, as a check on our implementation, before moving to real backgrounds.
Other methods. We also run:
- thresholded Oja (a single-pass streaming method);
- the kernel method of Seddik et al. (an entrywise nonlinearity applied to the covariance to suppress the noise eigenvalues), whose Gaussian-only proof the permuted-marginal cells probe directly;
- Fantope Projection and Selection, which is infeasible at $p=20{,}530$ because every iteration needs an eigendecomposition, so we run it only on prescreened subsets of about 2,000 features and report this restriction explicitly.
For subspace recovery on real noise there is no reported result found on Lacuna for any of these methods. Our numbers would be the first.
Fairness of comparison. All methods receive the same samples, the same sparsity budget, and the same wall-clock cap per run. Each gets two tuning regimes: an oracle $s$, and $s$ chosen by each of the three selectors. "Beating" a method means a phase transition at a lower rescaled sample size, by more than the width of the bootstrap confidence band, in the fully real cells.
Ablation. The key ablation is the two-by-two control itself. If the fully idealized cell ranks methods the same way the fully real cell does, then planting in real noise adds nothing over standard synthetic testing.
Why not just…?
Why not just benchmark sparse PCA features on TCGA classification? Accuracy cannot tell an estimator that found the true sparse direction from one that found a label-correlated noise direction. Across tasks, the reported spread is about 15 percentage points around means that differ from the majority baseline by 2.6 to 5 points. Differences between estimators would be lost in that noise. We still run this test as a secondary check. If the methods with better planted recovery do not also give better TCGA accuracy, we will report the disconnect rather than hide it.
Why not just simulate Gaussian data with a realistic covariance? This is a real threat. It is also exactly the "Gaussian, real correlations" cell. If that cell reproduces the fully real results, the honest conclusion is that correlation, not non-Gaussianity, drives the gap, and that cheap covariance-matched simulation suffices. That result would itself be useful guidance for the field.
Why not just use split-half stability or cross-validation as is? Split-half stability has never been calibrated against known truth for sparse PCA. Cross-validation has documented inconsistency in related high-dimensional problems. Planting is what lets us check whether either statistic tracks true error. If plain cross-validation selects $s$ as well as the stability rule, we will drop the stability rule from our recommendations.
Proposed Experiments
The first experiment checks the curve collapse. For each method and control cell, we sweep $n$ from 50 to several thousand and $s$ from 5 to 100. We plot the probability of support recovery and the angle error against $n/(\sigma^2 s\log p)$, with $\sigma^2$ computed from $\Sigma_{\text{ref}}$. The theory transfers if the curves from the four cells overlap.
The second experiment tracks ranking flips. We record the conditions under which the ordering of methods changes between the spherical Gaussian cell and the fully real cell. We focus on diagonal thresholding versus the restarted truncated power method, and on the kernel method under real marginals.
The third experiment tests sparsity selection. Measured by planted recovery error, we compare the stability rule against cross-validation and the false-discovery-rate selector at $n=50$, $p=20{,}530$. This is the starvation regime that the false-discovery-rate method left unbenchmarked.
The fourth experiment applies the calibrated stability statistic to the unplanted leading components of each real dataset. We then check its estimates against the TCGA downstream accuracies (MLP 71.19%, logistic regression 68.78%, majority 66.18%, as reported in the benchmark paper).
References
- Minimax Rates of Estimation for Sparse PCA in High Dimensions
- The TCGA Meta-Dataset Clinical Benchmark
- Combinatorial Sparse PCA Beyond the Spiked Identity Model
- Sparse PCA: Convex Relaxations, Algorithms and Applications
- Sparse PCA with False Discovery Rate Controlled Variable Selection
- Tighten after Relax: Minimax-Optimal Sparse PCA in Polynomial Time
- Fantope Projection and Selection: A near-optimal convex relaxation of sparse PCA
- Oja's Algorithm for Streaming Sparse PCA
- A Kernel Random Matrix-Based Approach for Sparse PCA
- Inconsistency of Cross-Validation for Structure Learning in Gaussian Graphical Models
- High-dimensional analysis of semidefinite relaxations for sparse principal components