Improving Neural Language Generation with Spectrum Control

Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, Quanquan GuICLR 2020 · OpenReview

TL;DR

State-of-the-art language models often suffer from "representation degeneration," a phenomenon where word embeddings are squeezed into a narrow, one-sided cone in vector space, severely limiting their expressiveness. This paper introduces Spectrum Control (SC), a novel method that explicitly manages the singular value distribution of the output embedding matrix. By ensuring word representations remain diverse and isotropic (spread out evenly), SC leads to significant performance gains in both language modeling and machine translation tasks.

The Representation Degeneration Problem

In neural language generation (NLG), models like Transformers map context to a hidden state and then multiply that state by an output embedding matrix to predict the next word. Ideally, these word embeddings should be "isotropic"—meaning they utilize the entire available vector space to maximize their expressive capacity.

However, researchers have found that these embeddings often degenerate. Instead of being spread out, they occupy a narrow "anisotropic cone."

Baseline Degeneration

Mathematically, this degeneration manifests as a "fast singular value decay": a few singular values are massive, while the rest drop off to near zero almost immediately. This restricts the model's ability to distinguish between different words and effectively model the conditional probability of the next token.

Methodology: Spectrum Control

The authors propose Spectrum Control (SC) to address this degeneration by directly intervening in the model's "spectrum" (its singular values).

  1. SVD Reparameterization: Instead of training the weight matrix $W$ directly, the method decomposes it using Singular Value Decomposition: $W = U\Sigma V^\top$. Here, $U$ and $V$ are orthonormal matrices, and $\Sigma$ is a diagonal matrix containing the singular values.
  2. Guided Training with Priors: The singular values in $\Sigma$ are forced to follow a "slow-decaying" prior distribution. The authors tested two types:
  3. Exponential Decay: Singular values drop off according to an exponential function.
  4. Polynomial Decay: Singular values drop off more gradually according to a power law.
  5. Regularization: To keep $U$ and $V$ orthonormal during training, a specific orthogonal regularization term is applied.
  6. Theoretical Framework: The authors provide a generalization error bound, proving that by controlling the "tail" of the singular value distribution, they can achieve a better trade-off between minimizing training loss and ensuring the model generalizes well to unseen data.

Key Insights and Ideas

  • The "Cone" is the Enemy: Standard Transformers produce a narrow cone of embeddings. After applying Spectrum Control, the 2D projections of these embeddings look like a dispersed cloud, indicating they are using the vector space much more efficiently.
Isotropic Embeddings with SC
  • Explicit vs. Implicit Control: Previous attempts to solve degeneration relied on implicit tricks, such as cosine similarity penalties. This paper argues that explicitly defining what the singular value distribution should look like is more robust and theoretically sound.
  • Scale Matters: A key finding is that exponential decay works better for smaller datasets (like WikiText-2), while polynomial decay (which is even more gradual) is superior for large-scale datasets (like WikiText-103).
  • Maintaining Power: Crucially, SC encourages diversity without sacrificing the modeling power of the Transformer. It finds the "sweet spot" between the model's need to focus on context and the need for a rich vocabulary representation.

Experimental Results

The authors evaluated their method on two primary tasks: Language Modeling (WikiText-2, WikiText-103) and Machine Translation (IWSLT 2014 German $\rightarrow$ English, WMT 14 English $\rightarrow$ German).

  • Language Modeling: On the large-scale WikiText-103 dataset, the SC-enhanced Transformer-XL achieved a test perplexity reduction of 0.8 over the standard state-of-the-art model.
  • Machine Translation: On the IWSLT 2014 task, the method improved the BLEU score by 1.5 points compared to the Transformer baseline, outperforming other recent techniques like adversarial training.
  • Isotropy Metrics: Using quantitative measures ($I_1$ and $I_2$), the authors showed that their method significantly increased the isotropy of the learned representations. For instance, the $I_1$ score increased from 0.24 to 0.63 in language modeling.

Limitations and Future Work

While highly effective, Spectrum Control introduces some computational overhead. Because the method involves SVD-related operations and extra regularization, it is roughly 1.17x to 1.24x slower than standard training and uses about 1.1x to 1.3x more memory.

The authors suggest that SC is orthogonal to other techniques like adversarial softmax training, making their combination a promising future direction. Additionally, while the paper touches on how word frequency relates to these embeddings, further research is needed to fully understand how SC specifically impacts the representations of "rare" versus "common" words.

Summary updated

Abstract

Recent Transformer-based models such as Transformer-XL and BERT have achieved huge success on various natural language processing tasks. However, contextualized embeddings at the output layer of these powerful models tend to degenerate and occupy an anisotropic cone in the vector space, which is called the representation degeneration problem. In this paper, we propose a novel spectrum control approach to address this degeneration problem. The core idea of our method is to directly guide the spectra training of the output embedding matrix with a slow-decaying singular value prior distribution through a reparameterization framework. We show that our proposed method encourages isotropy of the learned word representations while maintains the modeling power of these contextual neural models. We further provide a theoretical analysis and insight on the benefit of modeling singular value distribution. We demonstrate that our spectrum control method outperforms the state-of-the-art Transformer-XL modeling for language model, and various Transformer-based models for machine translation, on common benchmark datasets for these tasks.
BibTeX
@inproceedings{wang2020improving,
  author = {Lingxiao Wang and Jing Huang and Kevin Huang and Ziniu Hu and Guangtao Wang and Quanquan Gu},
  title = {{Improving Neural Language Generation with Spectrum Control}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=lzM-lymxEjc},
}
Figure