Introduction
Linear autoencoders (LAEs) are a fundamental building block in neural network theory, often viewed as a gradient-based method for performing Principal Component Analysis (PCA). While an unregularized LAE identifies the $k$-dimensional subspace spanned by the principal components, it fails to recover the specific principal directions themselves. This discrepancy arises from a mathematical symmetry: any invertible transformation applied to the internal representation leaves the reconstruction error unchanged. This paper provides a rigorous mathematical foundation for why $L_2$ regularization (weight decay) breaks these symmetries, forcing the network to align with the true principal axes.
Analyzing the Loss Landscape
The researchers investigate three distinct loss functions to understand how regularization alters the optimization surface:
- Unregularized ($\mathcal{L}$): Standard reconstruction error.
- Product-regularized ($\mathcal{L}_\pi$): Penalizes the composition of weights ($W_2 W_1$).
- Sum-regularized ($\mathcal{L}_\sigma$): Standard weight decay, penalizing the Frobenius norms of $W_1$ and $W_2$ separately.
Using tools from differential geometry and Morse Theory, the authors show that regularization "stiffens" the landscape. In the unregularized case, the global minimum is a "flat" manifold of equivalent solutions. Regularization reduces the symmetry group from all invertible matrices ($GL_k(\mathbb{R})$) to just orthogonal rotations ($O_k(\mathbb{R})$), creating distinct, isolated minima that correspond to the data's singular vectors.
The Transpose Theorem and Weight Alignment
A pivotal discovery in this work is the Transpose Theorem. For the sum-regularized loss ($\mathcal{L}_\sigma$), the authors prove that at every critical point, the encoder and decoder weights must be transposes of each other ($W_1 = W_2^T$). This symmetry is not enforced by architecture but emerges naturally from the optimization process.
This result has significant implications for computational neuroscience. The "weight transport problem" asks how biological neurons might communicate the transpose of forward weights for learning. The authors suggest that local reconstruction objectives combined with weight decay provide a biologically plausible mechanism for achieving this alignment.
Key Results and Empirical Validation
The theoretical framework was validated using both synthetic data and the MNIST dataset. On MNIST, the sum-regularized LAE successfully recovered "eigen-digits"—the actual principal components—whereas unregularized models produced noisy, uninterpretable weight patterns.
Furthermore, the authors demonstrate that the "heights" of saddle points in the loss landscape are directly tied to the eigenvalues of the data's covariance matrix. This topological perspective allows for a deeper understanding of why gradient descent reliably finds global minima in these regularized systems.
Conclusion
By mapping the loss landscapes of LAEs, this paper bridges the gap between neural network training and classical statistical methods like PCA. The findings highlight that regularization is not merely a tool for preventing overfitting but a fundamental mechanism for symmetry breaking and weight alignment. While currently limited to linear models, this topological approach offers a promising path toward understanding the complex landscapes of deep, non-linear networks.
Summary updated