Embracing the Blessing of Dimensionality in Factor Models

Quefeng Li, Guang Cheng, Jianqing Fan, Yuyan Wang

Abstract

Factor modeling is an essential tool for exploring intrinsic dependence structures among high-dimensional random variables. Much progress has been made for estimating the covariance matrix from a high-dimensional factor model. However, the blessing of dimensionality has not yet been fully embraced in the literature: much of the available data is often ignored in constructing covariance matrix estimates. If our goal is to accurately estimate a covariance matrix of a set of targeted variables, shall we employ additional data, which are beyond the variables of interest, in the estimation? In this paper, we provide sufficient conditions for an affirmative answer, and further quantify its gain in terms of Fisher information and convergence rate. In fact, even an oracle-like result (as if all the factors were known) can be achieved when a sufficiently large number of variables is used. The idea of utilizing data as much as possible brings computational challenges. A divide-and-conquer algorithm is thus proposed to alleviate the computational burden, and also shown not to sacrifice any statistical accuracy in comparison with a pooled analysis. Simulation studies further confirm our advocacy for the use of full data, and demonstrate the effectiveness of the above algorithm. Our proposal is applied to a microarray data example that shows empirical benefits of using more data.

Introduction

In modern statistical analysis, the "curse of dimensionality" is a well-known phenomenon where an increase in variables leads to noise accumulation and computational intractability. However, in the context of factor models—widely used in finance, genomics, and social sciences—high dimensionality can actually be a "blessing." Factor models assume that the behavior of thousands of observed variables is driven by a small number of latent, unobserved factors.

Standard practice typically involves isolating a subset of variables of interest ($s$) and estimating their covariance matrix using only that specific data. This paper challenges that convention, arguing that ignoring "auxiliary" variables ($p$) that share the same underlying factors is a missed opportunity. By incorporating these additional variables, researchers can significantly improve the estimation of the latent factors themselves, leading to more accurate covariance estimates for the original variables of interest.

Methodology

The authors contrast two primary approaches to covariance estimation:

  1. Conventional Method: Estimating factors and the resulting covariance matrix using only the $s$ variables of interest.
  2. Proposed Method: Utilizing all $p$ available variables (where $p \gg s$) to estimate the latent factors, then "plugging" these refined factor estimates back into the estimation for the $s$ variables.

Weighted Principal Component (WPC)

To handle the integration of diverse auxiliary data, the authors employ Weighted Principal Component analysis. Unlike standard PCA, WPC accounts for heteroscedasticity—varying noise levels across different variables. This is critical when auxiliary data may be noisier than the primary variables of interest.

Divide-and-Conquer (D&C) Algorithm

Processing massive datasets presents a computational bottleneck, as standard factor estimation often involves operations with $O(p^3)$ complexity. To alleviate this, the authors propose a Divide-and-Conquer algorithm. The $p$ variables are split into $M$ smaller groups; factor estimation is performed in parallel across these groups, and the results are subsequently averaged. This approach maintains the statistical benefits of the full dataset while drastically reducing the time required for computation.

Theoretical Insights

The paper provides a rigorous theoretical foundation for the "blessing of dimensionality" through Fisher information and convergence rate analysis.

  • Information Gain: The authors prove that as the total number of variables $p$ increases, the Fisher information regarding the latent factors grows. This suggests that auxiliary variables provide multiple "looks" at the same latent signal, allowing the collective signal to outweigh the individual noise of the extra variables.
  • Oracle Property: A key finding is that when $p$ is sufficiently large relative to $s$ and the number of time points $T$, the estimator achieves "oracle-level" accuracy—performing as if the true latent factors were known exactly.
  • Efficiency of D&C: Theoretically, the D&C approach does not sacrifice accuracy. As long as the number of groups $M$ does not grow too rapidly, the averaged factor estimates achieve the same convergence rate as a pooled analysis of the entire dataset.

Empirical Results

Simulation Studies

In controlled simulations where $s=50$ and $p=2000$, the proposed method reduced covariance matrix estimation error by approximately 25-30% compared to the conventional method. Furthermore, the D&C algorithm demonstrated significant scalability; in tests with $p \approx 6000$, the D&C method was 9 times faster than the pooled analysis while maintaining nearly identical error rates.

Real-World Application: Microarray Data

The method was applied to a microarray dataset (GSE22255) containing gene expression data from 40 subjects (20 with ischemic stroke and 20 controls). The researchers used Linear Discriminant Analysis (LDA) to classify patients. The results showed that "LDA-2"—which used all available genes to estimate the underlying factors—consistently achieved lower misclassification rates than "LDA-1," which relied only on a subset of differentially expressed genes.

Conclusion

This research demonstrates that in high-dimensional factor modeling, more data is indeed better. By embracing the blessing of dimensionality and utilizing auxiliary variables, researchers can achieve higher precision in covariance estimation. The proposed Divide-and-Conquer algorithm ensures that this accuracy does not come at the cost of computational feasibility. Future research may further refine these methods by exploring "smart" grouping strategies within the D&C framework or investigating the sensitivity of these models to incorrectly specified numbers of factors.

Summary updated

Abstract

Factor modeling is an essential tool for exploring intrinsic dependence structures among high-dimensional random variables. Much progress has been made for estimating the covariance matrix from a high-dimensional factor model. However, the blessing of dimensionality has not yet been fully embraced in the literature: much of the available data are often ignored in constructing covariance matrix estimates. If our goal is to accurately estimate a covariance matrix of a set of targeted variables, shall we employ additional data, which are beyond the variables of interest, in the estimation? In this article, we provide sufficient conditions for an affirmative answer, and further quantify its gain in terms of Fisher information and convergence rate. In fact, even an oracle-like result (as if all the factors were known) can be achieved when a sufficiently large number of variables is used. The idea of using data as much as possible brings computational challenges. A divide-and-conquer algorithm is thus proposed to alleviate the computational burden, and also shown not to sacrifice any statistical accuracy in comparison with a pooled analysis. Simulation studies further confirm our advocacy for the use of full data, and demonstrate the effectiveness of the above algorithm. Our proposal is applied to a microarray data example that shows empirical benefits of using more data. Supplementary materials for this article are available online.
BibTeX
@article{li2016embracing,
  author = {Quefeng Li and Guang Cheng and Jianqing Fan and Yuyan Wang},
  title = {{Embracing the Blessing of Dimensionality in Factor Models}},
  journal = {Journal of the American Statistical Association},
  year = {2016},
  doi = {10.1080/01621459.2016.1256815},
}
Figure