Improving Neural Language Models with Weight Norm Initialization and Regularization

Christian Herold, Yingbo Gao, Hermann NeyEMNLP 2018

TL;DR

This paper introduces novel techniques for initializing and regularizing the weight norms of embedding and projection matrices in neural language models (NLMs). By initializing weight norms based on word frequencies and applying a specific regularization loss, they achieve state-of-the-art perplexity improvements on standard language modeling benchmarks.

What Problem Does This Paper Address?

Neural language models (NLMs) are crucial for many NLP tasks, including machine translation and speech recognition. A core component of these models involves mapping words into continuous vector spaces using embedding and projection matrices. The authors observe that in well-trained NLMs, the norms of these word vectors tend to correlate with word frequencies – more frequent words have larger norm vectors. This observation suggests that standard initialization strategies (like random initialization) might not be optimal, and that the norms of these vectors could be a valuable signal to leverage. Furthermore, they aim to improve the generalization ability of these models, a common challenge in NLP, by specifically regularizing these important weight matrices.

Methods

The paper proposes two main contributions:

  • Weight Norm Initialization: Instead of random initialization, they initialize the norms of the weight vectors (specifically, the rows of the embedding and projection matrices) with scaled logarithms of word counts. To achieve this, they reparameterize the weight vectors into a magnitude (norm) and a direction. The norm is then initialized using the scaled log word counts, while the direction is sampled from a uniform distribution. They also introduce an adaptive gradient strategy to anneal the updates to these weight norms over time.
  • Weight Norm Regularization: They introduce a novel regularization loss term that specifically penalizes the deviation of the weight vector norms from a target value ($\nu$). This is applied to the embedding and projection matrices. This approach differs from traditional weight decay or L2 regularization, which affect all weights equally, by focusing specifically on the norms of these critical word representation vectors.

Datasets

The experiments were conducted on two standard language modeling datasets:

  • Penn Treebank (PTB): A smaller dataset with approximately 888k training tokens and a vocabulary size of 10k.
  • WikiText-2 (WT2): A larger dataset with approximately 2.1M training tokens and a vocabulary size of 33k.

Key Insights and Ideas

  • Word Norms Mirror Word Frequencies: The most striking insight is the strong correlation between the norms of learned word vectors and the logarithm of word counts. This suggests that NLMs implicitly learn to assign larger magnitudes to more frequent words, likely to better capture their statistical importance.
  • Frequency-Based Initialization is Beneficial: Leveraging this frequency-norm relationship for initialization significantly improves perplexity, especially in the early stages of training. This indicates that providing the model with a more informed starting point for word vector magnitudes can accelerate learning.
  • Targeted Regularization Improves Generalization: Applying regularization specifically to the norms of embedding and projection matrices, rather than all weights, leads to better generalization. This suggests that these matrices are particularly prone to overfitting, and that controlling their norm distribution is key.
  • Soft Regularization is Preferred: The authors opt for a soft regularization approach (penalizing deviation from a target norm) over hard-fixing norms. This is because they observe that NLMs naturally learn diverse norms for words with different counts, and hard constraints might hinder this beneficial diversity.

Key Results

  • Early Epoch Improvement: Weight norm initialization alone achieved a significant perplexity reduction of around 10% in the first epoch on both PTB and WT2 datasets.
  • State-of-the-Art Performance: By combining weight norm initialization and regularization, the authors achieved state-of-the-art results, improving perplexity by 1.28 ppl on PTB and 1.32 ppl on WT2 compared to previous best results (without dynamic evaluation).
  • Specific Improvements:
    • On Penn Treebank, their model achieved a test perplexity of 53.16, down from a baseline of 54.44.
  • On WikiText-2, their model achieved a test perplexity of 60.13, down from a baseline of 61.45.
  • No Parameter Increase: These improvements were achieved without increasing the number of trainable parameters or slowing down training.

Limitations and Future Work

  • Initialization Strategy: The authors suggest exploring smoothing techniques on word counts for initialization and investigating the effects of such smoothed initializations.
  • Norm Constraint Uniformity: They propose to examine altering the loss function to regularize weight norms differently based on word counts, rather than applying a uniform constraint.
  • Beyond Norms: A more exciting future direction is to study the pairwise inner products (angles) between word vectors, not just their norms, to understand and leverage angular differences.
  • Application to NMT: The authors plan to extend their regularization and initialization techniques to neural machine translation (NMT), particularly to transformer architectures where embedding and projection matrices are often tied.

Summary updated

BibTeX
@inproceedings{herold2018improving,
  author = {Christian Herold and Yingbo Gao and Hermann Ney},
  title = {{Improving Neural Language Models with Weight Norm Initialization and Regularization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure