Regularized linear autoencoders recover the principal components, eventually

Xuchan Bao, James Lucas, Sushant Sachdeva, Roger B. Grosse

Abstract

Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs). We show that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. We analyze two such regularization schemes: non-uniform $\ell_2$ regularization and a deterministic variant of nested dropout [Rippel et al, ICML' 2014]. Though both regularization schemes converge to the optimal representation, we show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We show that the inefficiency of learning the optimal representation is not inevitable -- we present a simple modification to the gradient descent update that greatly speeds up convergence empirically.

Introduction

In unsupervised learning, the goal is often to learn "meaningful" representations. For a linear autoencoder (LAE), the gold standard is Principal Component Analysis (PCA)—specifically, the ordered, axis-aligned principal components of the data. However, standard LAEs suffer from rotational symmetry: any invertible rotation between the encoder and decoder preserves the reconstruction loss. Consequently, while the model finds the correct subspace, the orientation within that subspace is typically arbitrary.

Symmetry Breaking and Ill-Conditioning

To recover the true principal components, researchers often employ regularization schemes like non-uniform $\ell_2$ regularization or nested dropout. These methods break rotational symmetry by penalizing latent dimensions differently, theoretically forcing the model to align its axes with the data's principal components.

Optimization paths in the loss landscape showing the trajectory toward global minima for different regularization schemes.

However, this symmetry breaking is mathematically "weak." The authors' Hessian analysis reveals that as the number of latent dimensions increases, the loss landscape develops a "narrow valley" (as visualized in Figure 1). This leads to severe ill-conditioning, where the force driving the model toward the correct subspace is much stronger than the force rotating it into the correct alignment. In practice, this means the model finds the right subspace quickly but takes an eternity to align its axes.

Rotation Augmented Gradient (RAG)

The authors propose a novel solution: the Rotation Augmented Gradient (RAG) update. Instead of relying on passive regularization to nudge the weights, RAG adds a specific term to the gradient update that explicitly accounts for the rotation of the latent space. By performing a small rotation at every step, RAG decouples the task of finding the subspace from the task of aligning the axes, effectively bypassing the ill-conditioned landscape created by standard regularization.

Key Results

Experiments on both synthetic datasets and MNIST demonstrate that RAG achieves axis-alignment orders of magnitude faster than traditional methods.

Comparison of (a) axis-alignment and (b) subspace convergence. While all methods find the subspace quickly, only RAG (rotation) achieves rapid axis-alignment.

As shown in Figure 3, while the reconstruction error (subspace distance) drops rapidly for all methods, the actual representation remains unaligned for thousands of epochs in standard regularized models. RAG, however, maintains consistent and fast convergence. Furthermore, while the convergence time for regularized LAEs grows quadratically with the number of latent dimensions, RAG remains efficient even in high-dimensional settings.

Conclusion

This work demonstrates that the slow recovery of principal components in LAEs is a result of fundamental ill-conditioning in the regularized loss landscape. By introducing the RAG update, the authors show that this inefficiency is not inevitable. These insights into "weak symmetry breaking" may provide a path forward for understanding and improving representation learning in more complex, non-linear models like Variational Autoencoders.

Summary updated

Abstract

Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs). We show that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. We analyze two such regularization schemes: non-uniform $\ell_2$ regularization and a deterministic variant of nested dropout [Rippel et al, ICML' 2014]. Though both regularization schemes converge to the optimal representation, we show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We show that the inefficiency of learning the optimal representation is not inevitable -- we present a simple modification to the gradient descent update that greatly speeds up convergence empirically.
BibTeX
@inproceedings{bao2020regularized,
  author = {Xuchan Bao and James Lucas and Sushant Sachdeva and Roger B. Grosse},
  title = {{Regularized linear autoencoders recover the principal components, eventually}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=8UGotpUe9jM},
}
Figure