On the Softmax Bottleneck of Recurrent Language Models

Dwarak Govind Parthiban, Yongyi Mao, Diana Inkpen

TL;DR

This paper challenges the widely held belief that the "softmax bottleneck"—the mathematical inability of standard softmax layers to produce high-rank probability matrices—is the primary factor limiting recurrent language models. Through extensive empirical testing, the authors demonstrate that a high-rank log-probability matrix is neither necessary nor sufficient for superior performance, suggesting that the success of models like Mixture of Softmaxes (MoS) may stem from increased model capacity and regularization rather than just "breaking" the bottleneck.

What Problem Does This Paper Address?

In neural language modeling, we represent the probability of the next word given a context using a softmax layer. In 2018, researchers (Yang et al.) proved a theoretical limitation: the rank of the resulting log-probability matrix ($\log P$) is bounded by the word embedding dimension ($d$). Because natural language is highly diverse and context-dependent, it was hypothesized that this "softmax bottleneck" prevents models from capturing complex distributions.

Following this theory, several high-performing architectures like Mixture of Softmaxes (MoS), SigSoftmax (SS), and LMS-PLIF were proposed specifically to increase the rank of the $\log P$ matrix. The community generally accepted that "higher rank equals better perplexity." This paper investigates whether that correlation actually holds true or if we’ve been attributing performance gains to the wrong mechanism.

Methods

The authors take a rigorous, empirical approach to debunking the rank-performance myth:

  1. Generalized SigSoftmax (GSS): They created a new family of functions that allow them to "dial" the rank of the $\log P$ matrix up or down by adjusting specific parameters ($c$ and $k$). This allowed them to test a wide variety of ranks within the same model architecture.
  2. Systematic Reproduction: They reproduced several state-of-the-art models (SS, LMS-PLIF, MoS) on a level playing field using the same optimization algorithm (ET-ASGD) to ensure fair comparison.
  3. Rank Analysis: They calculated the rank of the $\log P$ matrix using Singular Value Decomposition (SVD). Crucially, they introduced the $\epsilon$-effective rank—a metric that looks at how many singular values are actually significant—to see if standard rank calculations were overestimating the "true" diversity of the models.
  4. Downstream Evaluation: Beyond just perplexity (how well the model predicts the next word), they evaluated the quality of the learned word embeddings on 13 different word similarity benchmarks.

Key Insights and Ideas

The Correlation is Weak

The Pearson correlation coefficient between the rank of the $\log P$ matrix and test perplexity was only around 0.5. In some cases, excluding outliers dropped this correlation as low as 0.21. This suggests rank is a poor predictor of model quality.

Correlation between rank and test perplexity on PTB and WT2 datasets.

Capacity vs. Rank in MoS

While MoS does produce high-rank matrices, it also significantly increases the number of parameters (capacity). The authors show that as you increase the number of mixtures ($K$), performance improves alongside capacity, but you can also increase rank without improving performance by changing hyperparameters like dropout.

Impact of mixture count and parameter count on model perplexity.
  • The LMS-PLIF Paradox: The LMS-PLIF model achieves excellent perplexity despite having a relatively low rank (comparable to the baseline softmax). This strongly suggests its performance comes from an "implicit regularization" effect rather than breaking the bottleneck.
  • Rank Doesn't Help Embeddings: High-rank models did not necessarily produce better word embeddings for downstream tasks. In fact, the Mixture of Contexts (MoC) model—which does not break the softmax bottleneck—often produced better embeddings than the high-rank MoS.
  • The "Press Rank" Illusion: Standard methods for calculating rank (like the Press et al. approach) may be overestimating the effective rank. The authors found that singular values decay extremely rapidly in all models, meaning most of the "high rank" is concentrated in very low-variance dimensions.

Key Results

  • Perplexity vs. Rank: In experimental results, the SigSoftmax (SS) model showed a massive rank (4979) but actually performed worse than the baseline on PTB. Meanwhile, LMS-PLIF had a much lower rank (580) but significantly better perplexity.
  • Statistical Significance: On the WT2 dataset, while SS and GSS were statistically better than the baseline, the improvements were "minimal" compared to MoS.
  • Fair Comparison: When the authors controlled for parameter count, the performance gap between MoS and a well-tuned Softmax model narrowed significantly (p-value of 0.62), suggesting that much of MoS's advantage comes from its architecture and capacity rather than the mathematical properties of the mixture.

Limitations and Future Work

The authors acknowledge that while they have cast doubt on the importance of rank, they haven't fully isolated every factor. Specifically:

  • Inductive Bias: It remains possible that the structure of mixtures provides a helpful inductive bias that is separate from both rank and raw parameter count.
  • Hyperparameter Sensitivity: Language models are notoriously sensitive to hyperparameters (like dropout and batch size). While the authors tried to be fair, different architectures might "prefer" different settings.
  • Future Direction: The paper suggests future research should focus on why these "bottleneck-breaking" layers act as good regularizers or how they affect the model's inductive bias, rather than chasing higher mathematical rank.

Summary updated

Abstract

Recent research has pointed to a limitation of word-level neural language models with softmax outputs. This limitation, known as the softmax bottleneck refers to the inability of these models to produce high-rank log probability (log P) matrices. Various solutions have been proposed to break this bottleneck, including Mixture of Softmaxes, SigSoftmax, and Linear Monotonic Softmax with Piecewise Linear Increasing Functions. They were reported to offer better performance in terms of perplexity on test data. A natural perception from these results is a strong positive correlation between the rank of the log P matrix and the model's performance. In this work, we show via an extensive empirical study that such a correlation is fairly weak and that the high-rank of the log P matrix is neither necessary nor sufficient for better test perplexity. Although our results are empirical, they are established in part via the construction of a rich family of models, which we call Generalized SigSoftmax. They are able to create diverse ranks for the log P matrices. We also present an investigation as to why the proposed solutions achieve better performance.
BibTeX
@inproceedings{parthiban2021softmax,
  author = {Dwarak Govind Parthiban and Yongyi Mao and Diana Inkpen},
  title = {{On the Softmax Bottleneck of Recurrent Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17608},
}
Figure