Data Noising as Smoothing in Neural Network Language Models

Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, Andrew Y. Ng

Data Noising as Smoothing in Neural Network Language Models

TL;DR

This paper bridges the gap between "old-school" statistical language modeling and modern deep learning by demonstrating that adding noise to input data is mathematically equivalent to n-gram smoothing. By leveraging this connection, the authors develop advanced noising schemes—inspired by classical techniques like Kneser-Ney smoothing—that significantly improve the performance of Recurrent Neural Networks (RNNs) in language modeling and machine translation.

The Challenge of Data Sparsity

Language models face a fundamental "data sparsity" problem: because the number of possible word combinations is astronomical, even massive datasets contain only a tiny fraction of all valid sequences. While computer vision researchers have long used data augmentation (like flipping or cropping images) to help models generalize, discrete text is much harder to "distort" meaningfully without destroying its semantic structure.

Traditional n-gram models solved this sparsity issue using smoothing—mathematical tricks to reassign probability mass from seen sequences to unseen ones. However, neural networks (like LSTMs) typically rely on dropout (randomly turning off neurons) to prevent overfitting. The authors argue that standard dropout ignores the specific sequential structure of language, and they set out to find a way to apply the wisdom of classical smoothing directly to the inputs of neural networks.

Methods: Bridging Noising and Smoothing

The authors propose and analyze several "noising" primitives where words in the training data are randomly replaced during the training process. Crucially, they provide a mathematical proof showing that simple unigram noising in an RNN is equivalent to linear interpolation smoothing in a bigram model. This theoretical bridge allows them to "port" advanced smoothing logic into the world of neural networks.

Mapping of noising schemes to classical smoothing analogues

As summarized in the table above, the authors explore several specific noising strategies:

  1. Blank Noising (Interpolation): Replacing a word with a placeholder (e.g., _). This forces the model to rely more heavily on the surrounding context.
  2. Unigram Noising (Absolute Discounting): Replacing a word with a random word sampled from the overall vocabulary frequency.
  3. Kneser-Ney (KN) Noising: This is the paper's most sophisticated contribution. Instead of a fixed replacement probability, they use an adaptive noising probability ($\gamma$).
  4. Words that appear in many different contexts (high "fertility") are noised more often.
  5. The replacement word is chosen based on how many different words it typically follows, rather than just its raw frequency.

Key Insights

  • Noising as Data Augmentation: Noising isn't just "breaking" the data; it's a targeted form of data augmentation that creates "pseudo-counts" for sequences the model hasn't seen yet.
  • The "Sticky Pair" Problem: Some words almost always appear together (like "Humpty Dumpty"). Standard noising might break these up too often. Adaptive KN-noising handles this by looking at mutual information—it avoids "backing off" to simpler models when the specific word pair is highly informative.
  • Efficiency over Dropout: Unlike advanced dropout techniques (like Monte Carlo dropout) that require multiple passes at test time to get a good prediction, noising only happens during training. At test time, the model runs at standard speed on uncorrupted text.
  • Smoothing the Output: Empirical analysis showed that noising forces the model's internal probability distributions to become "flatter" (higher entropy), which is exactly the intended effect of classical smoothing.

Key Results

The researchers evaluated their methods on three standard benchmarks: the Penn Treebank (PTB) for word-level language modeling, Text8 to test scalability, and IWSLT 2015 for English-German machine translation.

Validation and training perplexity over epochs for unnoised vs. noised models

The empirical results clearly demonstrate the regularizing power of these techniques. As seen in the learning curves above, models trained with noising (blue lines) successfully avoid the sharp overfitting observed in unnoised baselines (red lines).

Specifically:

  • Language Modeling: On the Penn Treebank, the Bigram Kneser-Ney noising scheme reduced test perplexity (a measure of how "surprised" the model is by new text) from 77.5 (baseline with dropout) to 73.4. This matched the state-of-the-art at the time while being computationally cheaper.
  • Machine Translation: Adding noising to a sequence-to-sequence model improved the BLEU score (translation quality) by +1.4 points over the baseline.
  • Robustness: The benefits were consistent across both medium and large model architectures, proving that noising is a general-purpose regularizer.

Limitations and Future Work

The authors primarily focused on bigram-level analogues (looking at the relationship between two adjacent words). While RNNs theoretically capture much longer dependencies, the specific noising schemes developed here don't explicitly account for trigrams or higher-order structures.

Additionally, while the paper shows that noising works well for English and German, future work could explore how these techniques perform in low-resource languages or languages with very different morphological structures. Finally, while this research focused on LSTMs, the fundamental principles of data noising as smoothing are highly relevant to modern architectures like Transformers.

Summary updated

Abstract

Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequence-level settings such as language modeling. In this paper, we derive a connection between input noising in neural network language models and smoothing in n-gram models. Using this connection, we draw upon ideas from smoothing to develop effective noising schemes. We demonstrate performance gains when applying the proposed schemes to language modeling and machine translation. Finally, we provide empirical analysis validating the relationship between noising and smoothing.
BibTeX
@inproceedings{xie2017data,
  author = {Ziang Xie and Sida I. Wang and Jiwei Li and Daniel Lévy and Aiming Nie and Dan Jurafsky and Andrew Y. Ng},
  title = {{Data Noising as Smoothing in Neural Network Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=DQKPmxr6SZJ},
}
Figure