Data Noising as Smoothing in Neural Network Language Models
TL;DR
This paper bridges the gap between "old-school" statistical language modeling and modern deep learning by demonstrating that adding noise to input data is mathematically equivalent to n-gram smoothing. By leveraging this connection, the authors develop advanced noising schemes—inspired by classical techniques like Kneser-Ney smoothing—that significantly improve the performance of Recurrent Neural Networks (RNNs) in language modeling and machine translation.
The Challenge of Data Sparsity
Language models face a fundamental "data sparsity" problem: because the number of possible word combinations is astronomical, even massive datasets contain only a tiny fraction of all valid sequences. While computer vision researchers have long used data augmentation (like flipping or cropping images) to help models generalize, discrete text is much harder to "distort" meaningfully without destroying its semantic structure.
Traditional n-gram models solved this sparsity issue using smoothing—mathematical tricks to reassign probability mass from seen sequences to unseen ones. However, neural networks (like LSTMs) typically rely on dropout (randomly turning off neurons) to prevent overfitting. The authors argue that standard dropout ignores the specific sequential structure of language, and they set out to find a way to apply the wisdom of classical smoothing directly to the inputs of neural networks.
Methods: Bridging Noising and Smoothing
The authors propose and analyze several "noising" primitives where words in the training data are randomly replaced during the training process. Crucially, they provide a mathematical proof showing that simple unigram noising in an RNN is equivalent to linear interpolation smoothing in a bigram model. This theoretical bridge allows them to "port" advanced smoothing logic into the world of neural networks.
As summarized in the table above, the authors explore several specific noising strategies:
- Blank Noising (Interpolation): Replacing a word with a placeholder (e.g.,
_). This forces the model to rely more heavily on the surrounding context. - Unigram Noising (Absolute Discounting): Replacing a word with a random word sampled from the overall vocabulary frequency.
- Kneser-Ney (KN) Noising: This is the paper's most sophisticated contribution. Instead of a fixed replacement probability, they use an adaptive noising probability ($\gamma$).
- Words that appear in many different contexts (high "fertility") are noised more often.
- The replacement word is chosen based on how many different words it typically follows, rather than just its raw frequency.
Key Insights
- Noising as Data Augmentation: Noising isn't just "breaking" the data; it's a targeted form of data augmentation that creates "pseudo-counts" for sequences the model hasn't seen yet.
- The "Sticky Pair" Problem: Some words almost always appear together (like "Humpty Dumpty"). Standard noising might break these up too often. Adaptive KN-noising handles this by looking at mutual information—it avoids "backing off" to simpler models when the specific word pair is highly informative.
- Efficiency over Dropout: Unlike advanced dropout techniques (like Monte Carlo dropout) that require multiple passes at test time to get a good prediction, noising only happens during training. At test time, the model runs at standard speed on uncorrupted text.
- Smoothing the Output: Empirical analysis showed that noising forces the model's internal probability distributions to become "flatter" (higher entropy), which is exactly the intended effect of classical smoothing.
Key Results
The researchers evaluated their methods on three standard benchmarks: the Penn Treebank (PTB) for word-level language modeling, Text8 to test scalability, and IWSLT 2015 for English-German machine translation.
The empirical results clearly demonstrate the regularizing power of these techniques. As seen in the learning curves above, models trained with noising (blue lines) successfully avoid the sharp overfitting observed in unnoised baselines (red lines).
Specifically:
- Language Modeling: On the Penn Treebank, the Bigram Kneser-Ney noising scheme reduced test perplexity (a measure of how "surprised" the model is by new text) from 77.5 (baseline with dropout) to 73.4. This matched the state-of-the-art at the time while being computationally cheaper.
- Machine Translation: Adding noising to a sequence-to-sequence model improved the BLEU score (translation quality) by +1.4 points over the baseline.
- Robustness: The benefits were consistent across both medium and large model architectures, proving that noising is a general-purpose regularizer.
Limitations and Future Work
The authors primarily focused on bigram-level analogues (looking at the relationship between two adjacent words). While RNNs theoretically capture much longer dependencies, the specific noising schemes developed here don't explicitly account for trigrams or higher-order structures.
Additionally, while the paper shows that noising works well for English and German, future work could explore how these techniques perform in low-resource languages or languages with very different morphological structures. Finally, while this research focused on LSTMs, the fundamental principles of data noising as smoothing are highly relevant to modern architectures like Transformers.
Summary updated