Improving Neural Language Modeling via Adversarial Training

Dilin Wang, ChengYue Gong, Qiang Liu

Abstract

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting. In this paper, we present a simple yet highly effective adversarial training mechanism for regularizing neural language models. The idea is to introduce adversarial noise to the output embedding layer while training the models. We show that the optimal adversarial noise yields a simple closed-form solution, thus allowing us to develop a simple and time efficient algorithm. Theoretically, we show that our adversarial mechanism effectively encourages the diversity of the embedding vectors, helping to increase the robustness of models. Empirically, we show that our method improves on the single model state-of-the-art results for language modeling on Penn Treebank (PTB) and Wikitext-2, achieving test perplexity scores of 46.01 and 38.07, respectively. When applied to machine translation, our method improves over various transformer-based translation baselines in BLEU scores on the WMT14 English-German and IWSLT14 German-English tasks.

Introduction

Large-scale neural language models (NLMs), such as those based on Recurrent Neural Networks (RNNs) or Transformers, are powerful tools for natural language processing but are notoriously prone to overfitting. With millions of parameters and high-dimensional discrete inputs, these models often memorize training data instead of learning generalizable patterns. While standard regularization techniques like dropout are helpful, a significant performance gap remains. This paper introduces an "embarrassingly simple" adversarial training mechanism that injects "worst-case" noise into the output embedding layer, significantly improving model robustness and performance with minimal computational overhead.

Adversarial MLE: A Simple Closed-Form Solution

The core of the proposed method is Adversarial MLE (Maximum Likelihood Estimation). Training is framed as a minimax game where an adversary calculates a small, "worst-case" perturbation for each word's embedding in the output (softmax) layer, and the model attempts to minimize its prediction error despite this noise.

A key contribution is Theorem 3.1, which proves that the optimal adversarial noise has a simple closed-form solution: it is the negative normalized vector of the model's current hidden state. This analytical solution eliminates the need for expensive inner-loop optimizations typical of adversarial training, making the algorithm highly efficient.

Theoretical Insights: Promoting Embedding Diversity

The adversarial mechanism acts as a "large-margin" classifier. By training the model to correctly identify words even when their embeddings are pushed toward neighbors, it learns more robust decision boundaries. Furthermore, this noise implicitly encourages embedding diversity. If word vectors are too similar, the adversarial noise makes them indistinguishable; to maintain accuracy, the model is forced to push embeddings further apart in the vector space.

Diversity and training dynamics of the proposed method compared to AWD-LSTM.

As shown in Figure 4, the proposed method increases the L2 distance between embeddings and maintains higher eigenvalues for the embedding matrix, indicating a more diverse and expressive representation space compared to the AWD-LSTM baseline.

Experimental Results

The method was evaluated on standard benchmarks including Penn Treebank (PTB), WikiText-2 (WT2), and machine translation tasks (WMT14, IWSLT14).

State-of-the-art results on the Penn Treebank dataset.

On PTB and WikiText-2, the method achieved then-state-of-the-art test perplexity scores of 46.01 and 38.07, respectively. In machine translation, it improved BLEU scores by approximately 1.1 points over Transformer baselines. Notably, training time remained nearly identical to standard MLE, demonstrating that robustness can be achieved without sacrificing efficiency.

Conclusion

This work presents a simple yet effective regularization technique for NLMs. By focusing on the output layer and deriving a closed-form solution for adversarial noise, the authors provide a practical tool for fighting overfitting. Future research may explore applying this diversity-promoting mechanism to other domains, such as generative models or reinforcement learning, where representation degeneration is a known challenge.

Summary updated

Abstract

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting. In this paper, we present a simple yet highly effective adversarial training mechanism for regularizing neural language models. The idea is to introduce adversarial noise to the output embedding layer while training the models. We show that the optimal adversarial noise yields a simple closed-form solution, thus allowing us to develop a simple and time efficient algorithm. Theoretically, we show that our adversarial mechanism effectively encourages the diversity of the embedding vectors, helping to increase the robustness of models. Empirically, we show that our method improves on the single model state-of-the-art results for language modeling on Penn Treebank (PTB) and Wikitext-2, achieving test perplexity scores of 46.01 and 38.07, respectively. When applied to machine translation, our method improves over various transformer-based translation baselines in BLEU scores on the WMT14 English-German and IWSLT14 German-English tasks.
BibTeX
@inproceedings{wang2019improving,
  author = {Dilin Wang and ChengYue Gong and Qiang Liu},
  title = {{Improving Neural Language Modeling via Adversarial Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=SJW312bObB},
}
Figure