OxLM: A Neural Language Modelling Framework for Machine Translation

Paul Baltescu, Phil Blunsom, Hieu HoangPrague Bull. Math. Linguistics 2014 · OpenReview

Abstract

Neural Language Models (NLMs) offer superior generalization capabilities compared to traditional $n$-gram models in machine translation (MT) by representing words in continuous vector spaces. However, the computational cost of normalizing probabilities over large vocabularies has historically hindered their integration into real-time decoding. OxLM is an open-source C++ framework designed to overcome these limitations. By employing class-based factorization and Noise Contrastive Estimation (NCE), OxLM provides a practical path for integrating high-performance neural models directly into standard MT decoders like Moses and cdec. This paper details the architectural optimizations that allow OxLM to achieve significant improvements in translation quality while maintaining manageable training and decoding times.

Introduction

In the pipeline of machine translation, the language model acts as the primary arbiter of fluency. While traditional $n$-gram models are computationally efficient, they are limited by data sparsity—the inability to accurately score word sequences not seen during training. NLMs address this by learning distributed representations, yet they introduce a "normalization bottleneck." Calculating the probability of a single word traditionally requires a summation over the entire vocabulary, making them prohibitively slow for the thousands of evaluations required during beam search decoding.

Methodology

OxLM implements a log-bilinear language model architecture, optimized for production-scale MT through several key techniques:

Class-based Factorization

To mitigate the $O(|V|)$ complexity of the output layer, OxLM utilizes class-based factorization. By partitioning the vocabulary into classes (e.g., via Brown clustering), the model first predicts a word class and then the specific word within that class. This reduces the per-word computation to approximately $O(\sqrt{|V|})$.

Noise Contrastive Estimation (NCE)

Training efficiency is addressed through NCE. Instead of calculating a full, normalized probability distribution, the model is trained to distinguish target words from a set of "noise" samples. This allows the model to learn unnormalized scores that closely approximate true probabilities, drastically reducing training time without significant loss in translation quality.

Hybrid Feature Integration

OxLM allows for the inclusion of "Direct $n$-gram features." This hybrid approach combines the generalization power of neural embeddings for rare words with the precise memorization of frequent phrases characteristic of classical $n$-gram models.

Decoding Optimizations

To facilitate integration with decoders like cdec and Moses, OxLM implements normalization caching. Since decoders frequently explore multiple hypotheses sharing the same prefix, caching these terms avoids redundant neural network passes, significantly improving throughput during beam search.

Experimental Setup

The framework was evaluated on French-to-English translation tasks using the Europarl-v7 and News-Commentary-v9 datasets. Models were trained on approximately 55 million tokens with a vocabulary of 57,782 words. Performance was measured using BLEU scores and decoding latency on the WMT14 (newstest2012) benchmark.

Results and Discussion

Translation Quality and Training Speed

The "DirectFactoredLM" configuration achieved a 0.7 BLEU point improvement over a highly optimized KenLM baseline. Crucially, the use of NCE with a single noise sample reduced training time from 34.1 hours to just 9.4 hours. A notable finding is that while NCE training resulted in higher perplexity, the impact on final BLEU scores was negligible, suggesting that MT performance is robust to the slight "blurring" of probability distributions introduced by NCE.

Decoding Performance

Despite optimizations, neural models remain more computationally intensive than $n$-gram models. OxLM's factored model required approximately 3.36 seconds per sentence, compared to 0.45 seconds for a standard $n$-gram model. While this represents a roughly 10x slowdown, it is a substantial improvement over unoptimized NLMs and makes neural integration feasible for high-quality research systems.

Conclusion

OxLM demonstrates that with clever architectural choices like NCE and class-based factorization, neural language models can be effectively integrated into large-scale MT decoders. While the inference overhead remains a challenge for real-time production, the framework provides a robust tool for researchers to push the boundaries of translation fluency through deep integration with existing decoding infrastructure.

Summary updated

BibTeX
@inproceedings{baltescu2014oxlm,
  author = {Paul Baltescu and Phil Blunsom and Hieu Hoang},
  title = {{OxLM: A Neural Language Modelling Framework for Machine Translation}},
  booktitle = {Prague Bull. Math. Linguistics},
  year = {2014},
  url = {https://openreview.net/forum?id=iaqsFMlUC_1},
}
Figure