Subspace Recovery in Winsorized PCA: Insights into Accuracy and Robustness

Sangil Han, Kyoowon Kim, Sungkyu JungAISTATS 2025

Presented at AISTATS 2025, "Subspace Recovery in Winsorized PCA" provides the first rigorous theoretical foundation for a common data-cleaning heuristic: clipping outliers. This technique, known as Winsorization, is frequently used in high-dimensional data analysis to protect against extreme values, but its impact on the accuracy of Principal Component Analysis (PCA) has remained largely unquantified until now.

The Problem: PCA’s "Glass Jaw"

Principal Component Analysis is a fundamental tool for dimension reduction, yet it is notoriously sensitive to outliers. A single extreme data point can significantly rotate the recovered subspace, leading to inaccurate results. While robust alternatives like Principal Component Pursuit exist, they are often computationally expensive and do not scale well to massive datasets.

Practitioners often resort to Winsorization (or clipping): if a data point's magnitude exceeds a radius $r$, it is projected back onto a ball of that radius. Despite its popularity in fields like differential privacy, the theoretical limits of this approach—specifically how much corruption it can handle and how the choice of $r$ affects recovery—were previously unknown.

Methods and Theoretical Contributions

The authors analyze Winsorized PCA (WPCA), a two-step process where data is first clipped to a radius $r$ and then subjected to standard PCA. By utilizing the framework of elliptical distributions (encompassing both Gaussian and heavy-tailed $t$-distributions), the paper derives several key mathematical guarantees:

  • Concentration Bounds: Guarantees on the proximity of the sample WPCA subspace to the true population subspace.
  • Breakdown Points: A new definition of "breakdown" for subspaces, distinguishing between "weak" drift and "strong" breakdown (where the recovered subspace becomes useless).
  • Perturbation Bounds: Formulas predicting the degree of subspace "wobble" based on the percentage of outliers.

Empirical Validation and the "Goldilocks" Radius

To validate their theory, the researchers conducted synthetic simulations using heavy-tailed ($t_3$) and light-tailed (Gaussian) distributions, as well as contaminated models with injected outliers.

Figure 6: Error in subspace recovery ($̂E\sinΘ$) as a function of clipping radius $r$ across different distributions and contamination levels (ϵ).

A central finding is the existence of a "Goldilocks" radius. As shown in the simulations, the choice of $r$ involves a critical trade-off: if $r$ is too small, essential signal information is lost (approaching Spherical PCA); if $r$ is too large, the estimator becomes vulnerable to noise and outliers. The results demonstrate that a non-zero, optimal radius exists that minimizes recovery error for both heavy and light-tailed data.

Key Insights

  1. High-Dimensional Optimality: WPCA achieves the minimax optimal rate of $O(\sqrt{p/n})$ for subspace recovery, making it as efficient as the best possible theoretical estimators even as the number of variables ($p$) grows.
  2. Resilience to Corruption: Unlike traditional PCA, which can be ruined by a single outlier, WPCA can tolerate a fixed proportion of outliers regardless of their extremity. The breakdown point scales with the "eigenvalue gap," providing significant resilience.
  3. Predictable Performance: The derived perturbation bounds allow researchers to predict exactly how many degrees a PCA vector might deviate given a known percentage of outliers.

Conclusion

This research transforms a popular "heuristic hack" into a principled statistical method. By providing a theoretical framework for Winsorization, the paper offers practitioners the mathematical assurance that clipping is not just a convenient pre-processing step, but a theoretically sound way to achieve robust and accurate subspace recovery in high-dimensional settings.

Summary updated

BibTeX
@inproceedings{han2025subspace,
  author = {Sangil Han and Kyoowon Kim and Sungkyu Jung},
  title = {{Subspace Recovery in Winsorized PCA: Insights into Accuracy and Robustness}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
}
Figure