No Double Descent in PCA: Training and Pre-Training in High Dimensions

Daniel Gedon, Antonio H. Ribeiro, Thomas B. SchönICLR 2023 · OpenReview

Abstract

With the recent body of work on overparameterized models the gap between theory and practice in contemporary machine learning is shrinking. While many of the present state-of-the-art models have an encoder-decoder architecture, there is little theoretical work for this model structure. To improve our understanding in this direction, we consider linear encoder-decoder models, specifically PCA with linear regression on data from a low-dimensional manifold. We present an analysis for fundamental guarantees of the risk and asymptotic results for isotropic data when the model is trained in a supervised manner. The results are also verified in simulations. Furthermore, we extend our analysis to the popular setting where parts of the model are pre-trained in an unsupervised manner by pre-training the PCA encoder with subsequent supervised training of the linear regression. We show that the overall risk depends on the estimates of the eigenvectors in the encoder and present a sample complexity requirement through a concentration bound. The results highlight that using more pre-training data decreases the overall risk only if it improves the eigenvector estimates. Therefore, we stress that the eigenvalue distribution determines whether more pre-training data is useful or not.
BibTeX
@inproceedings{gedon2023no,
  author = {Daniel Gedon and Antonio H. Ribeiro and Thomas B. Schön},
  title = {{No Double Descent in PCA: Training and Pre-Training in High Dimensions}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ieWqvOiKgz2},
}

Loading related content...

Figure