Large Margin Neural Language Model

Jiaji Huang, Yi Li, Wei Ping, Liang Huang

Abstract: We propose a large margin criterion for training neural language models. Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. However, we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation. The proposed method aims to enlarge the margin between the "good" and "bad" sentences in a task-specific sense. It is trained end-to-end and can be widely applied to tasks that involve re-scoring of generated text. Compared with minimum-PPL training, our method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation.

Introduction

Traditional neural language models (NLMs) are optimized to minimize perplexity (PPL), effectively learning to predict the next word in a sequence based on high-quality, "gold-standard" text. While this approach ensures the model assigns high probabilities to grammatical sentences, it often fails in downstream tasks like Automatic Speech Recognition (ASR) or Machine Translation (MT). In these scenarios, the NLM is used to re-score a list of candidate hypotheses. A model trained solely on PPL may inadvertently favor an incorrect hypothesis if it contains more frequent phrases, even if the overall sentence is less accurate than the ground truth.

To address this, we propose a "Large Margin" training criterion that shifts the focus from pointwise probability estimation to discriminative pairwise comparison. By explicitly training the model to distinguish between "good" and "bad" sentences, we provide it with the discriminative power necessary to push the score of the correct sentence significantly above those of incorrect alternatives.

Methodology: LMLM and rLMLM

We introduce two primary formulations for discriminative training:

  1. Large Margin Language Model (LMLM): This approach utilizes a hinge loss function. The model is trained to ensure that the log-likelihood of the ground-truth sentence is at least a predefined margin ($\tau$) higher than that of an imperfect hypothesis generated by the ASR or MT system.
  2. Ranking-loss-based LMLM (rLMLM): A more sophisticated variant that considers the relative quality of multiple hypotheses. It enforces a ranking where sentences with lower Word Error Rate (WER) or higher BLEU scores must maintain higher LM scores than lower-quality candidates by a specific margin.
Training loss curves showing the stability of the proposed method (b) compared to potentially unstable alternatives (a).

These models are fine-tuned from a pre-trained RNNLM, allowing them to leverage general linguistic knowledge while adapting to the specific error patterns of the target task. As shown in the training curves, the proposed formulation achieves stable convergence during this fine-tuning process.

Experimental Results

We evaluated our approach across three major benchmarks: the Wall Street Journal (WSJ) ASR dataset, a large-scale 10,000-hour noisy speech dataset, and the NIST OpenMT Chinese-to-English translation set.

  • ASR Performance: On the WSJ eval92 set, LMLM achieved a WER of 5.56, a significant improvement over the 7.69 baseline and the 6.59 achieved by a standard RNNLM. On the 10K-hour dataset, rLMLM reduced the WER from 20.90 to 19.49.
  • Machine Translation: In Chinese-to-English translation, rLMLM improved the BLEU score to 33.13, outperforming the baseline (31.52) and surpassing the standard RNNLM by nearly 1.0 point.

Analysis: The Perplexity Paradox

A key finding of this research is the "Perplexity Paradox": LMLM and rLMLM often result in higher (worse) perplexity than standard models, yet they perform significantly better on actual tasks. This confirms that PPL is a proxy metric that does not always correlate with real-world utility.

Distribution of the margin between ground-truth and hypothesis sentences. Plot (b) shows how LMLM (green) shifts the margin distribution to the right compared to standard RNNLM (blue).

Visualizing the margin distribution reveals why these models succeed. As seen in the histograms above, the Large Margin approach successfully shifts the probability mass, ensuring that the difference in log-probabilities between the correct sentence and the hypothesis is consistently larger. Furthermore, the Large Margin models showed a tighter, more linear correlation between LM scores and task metrics (like Word Accuracy) and helped correct the inherent length bias found in standard LMs.

Conclusion

By moving beyond perplexity minimization to a large margin criterion, we have developed a more effective way to train language models for re-scoring tasks. While the ranking-based approach introduces some additional computational complexity during training, the resulting gains in ASR and MT performance demonstrate the power of discriminative training in neural language modeling.

Summary updated

Abstract

We propose a large margin criterion for training neural language models. Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. However, we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation. The proposed method aims to enlarge the margin between the "good" and "bad" sentences in a task-specific sense. It is trained end-to-end and can be widely applied to tasks that involve re-scoring of generated text. Compared with minimum-PPL training, our method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation.
BibTeX
@inproceedings{huang2018large,
  author = {Jiaji Huang and Yi Li and Wei Ping and Liang Huang},
  title = {{Large Margin Neural Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=Bk4yjGGO-r},
}
Figure