Regularized linear autoencoders recover the principal components, eventually

Xuchan Bao, James Lucas, Sushant Sachdeva, Roger B. Grosse

TL;DR

While linear autoencoders (LAEs) can theoretically learn the same features as Principal Component Analysis (PCA), standard gradient descent is notoriously slow at "un-rotating" the latent space to find the actual principal components. This paper explains this slow convergence through Hessian analysis and introduces the Rotation Augmented Gradient (RAG), a novel update rule that explicitly speeds up the recovery of ordered, axis-aligned principal components.

The Problem with Linear Autoencoders

In the world of dimensionality reduction, PCA is the gold standard because it provides ordered, axis-aligned principal components. Linear Autoencoders (LAEs) are often used as a neural network alternative to PCA. However, there is a catch: while an unregularized LAE finds the correct subspace, it doesn't automatically find the individual principal components. Because of rotational symmetry, the encoder and decoder can "spin" the latent representation without changing the reconstruction loss.

Researchers have tried to fix this by adding regularization (like $\ell_2$ weight decay or "nested dropout"). While these methods eventually break the symmetry and recover the principal components, they are painfully slow. The authors tackle two fundamental questions:

  1. Why is the convergence to the optimal representation so slow?
  2. Can we design a better optimization rule to fix it?

Methods and Analysis

The authors analyze two existing regularization schemes and propose a new optimization strategy to overcome their limitations:

  1. Non-uniform $\ell_2$ Regularization: By applying different amounts of weight decay to different latent dimensions, the model is forced to prioritize certain dimensions, breaking the rotational symmetry.
  2. Deterministic Nested Dropout: A variant of dropout where units are dropped in a fixed order, forcing the network to pack the most important information into the earlier dimensions.
  3. Hessian Conditioning Analysis: The authors mathematically analyze the "flatness" of the loss landscape. They prove that the "valleys" corresponding to rotation are extremely narrow, leading to poor Hessian conditioning that worsens as the latent dimension ($k$) increases.
  4. Rotation Augmented Gradient (RAG): This is the paper's primary contribution. Instead of relying on a loss function to slowly nudge the weights, RAG introduces a specific term into the gradient update that explicitly "rotates" the latent space toward axis-alignment. Crucially, this rotation term is designed to be "loss-neutral"—it doesn't interfere with the model's ability to minimize reconstruction error.

Key Insights

Learning in a regularized LAE happens in a "Two-Stage" Learning Process. First, the model quickly finds the correct PCA subspace (moving in the "radial" direction in the loss landscape). Second, it very slowly rotates the axes to align with the principal components (moving along the "angular" direction in the valley).

Loss landscape trajectories showing the two-stage convergence process.

As illustrated in the loss landscape above, standard regularized methods (like non-uniform regularization) take a long, curved path along the valley floor. Ill-conditioning is the enemy here: the condition number of the Hessian grows at least quadratically with the latent dimension $k$. This means that for large-scale problems, standard gradient descent will essentially never reach the optimal axis-aligned representation.

RAG demonstrates that we don't always need a more complex loss function to achieve a specific representation; sometimes, modifying the update rule to account for known symmetries is more effective. Furthermore, RAG is closely related to the Generalized Hebbian Algorithm (GHA) but is more stable because its rotation term is mathematically decoupled from the reconstruction gradient.

Key Results

The paper evaluates these methods on MNIST and a high-dimensional synthetic dataset to test scalability.

Convergence comparison showing axis-alignment and subspace distance over epochs.
  • Speedup: On MNIST, RAG (labeled as "rotation" in the figures) reaches an axis-aligned solution significantly faster than non-uniform $\ell_2$ or nested dropout, as seen in the sharp drop in axis-alignment error.
  • Scalability: In synthetic tests, as the latent dimension $k$ increased, traditional regularized models failed to reach the target alignment within 50,000 epochs. RAG, however, scaled gracefully and maintained fast convergence.
  • Qualitative Success: Visualizations of the decoder weights (the "eigen-digits") show that RAG produces clear, distinct principal components early in training, whereas other methods produce blurry, mixed representations for a long time.

Limitations and Future Work

The most significant limitation is that the analysis and the RAG update rule are strictly for linear autoencoders. While LAEs are a vital theoretical playground, extending these insights to deep, non-linear architectures remains an open challenge. Additionally, RAG requires a custom update rule rather than just a standard loss function, which may make it slightly more cumbersome to implement in high-level deep learning frameworks.

Looking forward, the authors suggest that the idea of "rotation augmentation" could be viewed as an artificial gauge field, potentially opening doors to new optimization techniques for other models with known symmetries, such as GANs or Contrastive Learning.

Summary updated

Abstract

Our understanding of learning input-output relationships with neural nets has improved rapidly in recent years, but little is known about the convergence of the underlying representations, even in the simple case of linear autoencoders (LAEs). We show that when trained with proper regularization, LAEs can directly learn the optimal representation -- ordered, axis-aligned principal components. We analyze two such regularization schemes: non-uniform L2 regularization and a deterministic variant of nested dropout [Rippel et al, ICML' 2014]. Though both regularization schemes converge to the optimal representation, we show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We show that the inefficiency of learning the optimal representation is not inevitable -- we present a simple modification to the gradient descent update that greatly speeds up convergence empirically.
BibTeX
@inproceedings{bao2020regularized,
  author = {Xuchan Bao and James Lucas and Sushant Sachdeva and Roger B. Grosse},
  title = {{Regularized linear autoencoders recover the principal components, eventually}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=8UGotpUe9jM},
}
Figure