Autoencoders that don't overfit towards the Identity

Harald SteckNeurIPS 2020 · OpenReview

Introduction

Autoencoders (AEs) are designed to reproduce their input at the output. However, when used for prediction tasks like collaborative filtering, they often "cheat" by learning the identity function—simply copying inputs to outputs rather than learning useful feature interactions. While standard dropout denoising is widely believed to prevent this by forcing the model to predict hidden features from visible ones, this paper proves that it only acts as an L2 regularization and does not fully stop identity overfitting.

By analyzing linear autoencoders, the author demonstrates that "emphasized denoising"—upweighting the reconstruction error of dropped-out features—completely eliminates this overfitting and introduces a novel orthogonality constraint that improves recommendation accuracy.

Methodology

To achieve exact, analytic insights, the study focuses on Linear Autoencoders (LAEs) trained with least squares. The author deconstructs standard denoising, mathematically proving that standard dropout denoising (DLAE) is asymptotically equivalent to applying a specific, feature-adaptive L2 regularization. While it shrinks the identity function, it doesn't eliminate it.

To solve this, the paper introduces Emphasized Denoising (EDLAE). This variant upweights the reconstruction error of dropped-out features by a factor $a$, while uncorrupted features receive a lower weight $b$. By setting $b=0$ (Full Emphasis), the model ignores the reconstruction error of features that were not dropped out. The author derives a closed-form solution for full-rank models under this condition. For low-rank models, this creates a mathematical environment where the diagonal of the weight matrix (the identity component) is completely subtracted during training.

Key Theoretical Insights

  • Dropout $\neq$ No Identity: Standard dropout does not strictly prevent an autoencoder from learning the identity function; it merely applies L2 regularization. Given enough capacity, the model will still partially predict a feature from itself.
  • Why Dropout Beats Weight Decay: Even though dropout acts as L2 regularization, it is vastly superior to standard weight decay because it is invariant to model parameterization (e.g., matrix factorization) and automatically adapts to the variance of each specific feature.
  • The Counterintuitive Orthogonality Constraint: In low-rank models, preventing identity overfitting forces the latent embeddings of the encoder and decoder to be nearly orthogonal to each other. Surprisingly, highly "similar" features end up with dissimilar (orthogonal) embeddings to prevent their dot product from accidentally approximating the forbidden identity function.

Empirical Results

The models were evaluated on three standard implicit feedback collaborative filtering benchmarks: MovieLens 20 Million (ML-20M), Netflix Prize, and Million Song Data (MSD).

Table comparing model training results (including EDLAE and DLAE) across ML-20M, Netflix, and MSD datasets.
  • Quantitative Accuracy: EDLAE consistently outperformed standard DLAE, standard weight decay, and several deep non-linear baselines. As shown in the results table, on the ML-20M dataset, EDLAE achieved an nDCG@100 of 0.420, compared to DLAE (0.413) and standard weight decay (0.376).
  • Rank Robustness: Standard DLAE models start to lose accuracy as the matrix rank increases (around $k \approx 100$) because they begin to overfit to the identity. EDLAE maintains top accuracy across all matrix ranks, up to full rank.
  • Empirical Proof of Orthogonality: Analysis of the learned embeddings confirmed the theoretical math: the cosine similarity of latent embeddings for "similar" items is drastically reduced (pushed toward zero/orthogonality) in EDLAE compared to standard DLAE.

Limitations and Future Work

The primary limitation is theoretical: the mathematical derivations, closed-form solutions, and proofs are strictly limited to linear autoencoders to ensure tractability. However, the actual mechanism—stochastic emphasized denoising—is highly practical and can be readily applied to train deep, non-linear autoencoders, suggesting a strong direction for future empirical work in deep learning architectures.

Summary updated

Abstract

Autoencoders (AE) aim to reproduce the output from the input. They may hence tend to overfit towards learning the identity-function between the input and output, i.e., they may predict each feature in the output from itself in the input. This is not useful, however, when AEs are used for prediction tasks in the presence of noise in the data. It may seem intuitively evident that this kind of overfitting is prevented by training a denoising AE, as the dropped-out features have to be predicted from the other features. In this paper, we consider linear autoencoders, as they facilitate analytic solutions, and first show that denoising / dropout actually prevents the overfitting towards the identity-function only to the degree that it is penalized by the induced L2-norm regularization. In the main theorem of this paper, we show that the emphasized denoising AE is indeed capable of completely eliminating the overfitting towards the identity-function. Our derivations reveal several new insights, including the closed-form solution of the full-rank model, as well as a new (near-)orthogonality constraint in the low-rank model. While this constraint is conceptually very different from the regularizers recently proposed, their resulting effects on the learned embeddings are empirically similar. Our experiments on three well-known data-sets corroborate the various theoretical insights derived in this paper.
BibTeX
@inproceedings{steck2020autoencoders,
  author = {Harald Steck},
  title = {{Autoencoders that don't overfit towards the Identity}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=wrh_TI8hyJ8},
}
Figure