Recurrent Continuous Translation Models

Nal Kalchbrenner, Phil BlunsomEMNLP 2013 · OpenReview

Abstract

We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models (RCTMs) that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of state-of-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations.

Introduction

In the early 2010s, Statistical Machine Translation (SMT) was dominated by phrase-based models that relied on massive, discrete tables of word alignments. While effective, these models struggled with "sparsity"—if a specific phrase hadn't been seen exactly as written during training, the system was often lost. In this 2013 paper, Nal Kalchbrenner and Phil Blunsom introduced Recurrent Continuous Translation Models (RCTMs). This work was a pivotal step toward the Neural Machine Translation (NMT) revolution, proposing a way to translate by mapping sentences into a continuous mathematical space rather than relying on rigid, discrete alignments.

Methodology: The Convolutional Sentence Model

Traditional SMT systems viewed translation as a matching game between discrete units. This approach suffered from sparsity and the alignment bottleneck, where explicit decisions about word-to-word correspondence were computationally expensive and often failed to capture holistic meaning.

To address this, the authors developed the Convolutional Sentence Model (CSM). The CSM processes the source sentence using layers of convolutions to progressively build a representation, moving from individual words to n-grams, and finally to a single vector e representing the whole sentence. This allows the system to generalize to unseen phrases based on their semantic similarity to known ones.

The Convolutional Sentence Model (CSM) hierarchically processes a source sentence into a fixed-length continuous vector representation e.

Methodology: Translation Architectures

The RCTM architectures combine the CSM with a Recurrent Language Model (RLM) to generate the target sentence word-by-word. The paper introduces two specific variants for conditioning the generation on the source sentence:

  • RCTM I (Global Conditioning): The entire source sentence is compressed into one vector e. This vector is then fed into the RLM at every time step to guide the generation of the target words.
  • RCTM II (Localized Conditioning): Instead of one global vector, this model uses an "inverse" convolutional process to "unfold" the source representation into a sequence of vectors that matches the length of the target sentence. This allows the model to focus on different parts of the source information as it generates each target word—a precursor to modern attention mechanisms.
Architectures of RCTM I and RCTM II. RCTM I uses a global sentence vector, while RCTM II uses an 'inverse' convolutional process to provide localized conditioning.

Experimental Evaluation

The models were trained and evaluated on the WMT 2013 News Commentary English-French corpus. Key results include:

  • Massive Perplexity Drop: RCTM II achieved a perplexity that was 43% lower than the state-of-the-art IBM Model 2.
  • Sensitivity to Order: By testing the model on source sentences with randomly shuffled words, the authors proved that the CSM captures syntax and word order, not just "bag-of-words" meaning.
  • Competitive Rescoring: When used to rescore potential translations generated by traditional systems, the RCTM probabilities (using only two features) matched the performance of systems using twelve engineered features.
  • Qualitative Success: The model demonstrated an ability to maintain correct gender and number agreement in French (e.g., correctly choosing between le and la) based on the English source, despite having no explicit grammatical rules.

Conclusion

RCTMs demonstrated that continuous representations could effectively replace discrete phrase tables and explicit alignments. While they faced limitations like fixed-length constraints and the "bottleneck" of squeezing sentences into single vectors, they paved the way for the sequence-to-sequence models and attention mechanisms that define modern machine translation. Once trained, the probability of a translation can be computed using simple matrix-vector products, offering a path toward more efficient and semantically aware translation systems.

Summary updated

Abstract

We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations.
BibTeX
@inproceedings{kalchbrenner2013recurrent,
  author = {Nal Kalchbrenner and Phil Blunsom},
  title = {{Recurrent Continuous Translation Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=ByZ-MMf_WH},
}
Figure