Abstract
Transductive Ensemble Learning (TEL) is a novel approach for Neural Machine Translation (NMT) that leverages the availability of test data in offline scenarios. Unlike traditional ensembles that average model outputs during inference, TEL uses an ensemble to generate synthetic translations for the test set and then fine-tunes a single model on this data. This method achieves state-of-the-art performance on WMT English-German tasks, effectively overcoming the diminishing returns of standard ensembles while maintaining the inference speed of a single model.
1. Introduction
Traditional ensemble learning, which combines multiple models to improve prediction accuracy, is a staple in NMT. However, it suffers from three primary drawbacks: marginal gains for already strong models, diminishing returns as more models are added, and significantly increased inference latency. These issues make large-scale ensembles impractical for many real-world applications.
The authors propose Transductive Ensemble Learning (TEL) to address these challenges. TEL shifts the ensemble process from the inference phase to a transductive fine-tuning phase. This approach is particularly effective in "offline" settings—such as translating documents, books, or static databases—where the source text is known in advance.
2. Methodology
TEL operates in two distinct stages:
- Synthetic Dataset Construction: A set of $K$ pre-trained NMT models is used to translate the target test and validation sets. This results in a synthetic dataset where each source sentence is paired with $K$ different target translations.
- Transductive Fine-tuning: A single strong model (typically one of the $K$ models) is fine-tuned on this synthetic dataset. By optimizing its parameters to predict the various translations generated by the ensemble, the model distills the collective knowledge of the ensemble into a single set of parameters.
This approach allows for the use of "homogeneous" models—checkpoints from the same training run—which typically provide little benefit in traditional ensembles but prove effective under the TEL framework.
3. Results and Analysis
The authors evaluated TEL on high-profile benchmarks, including WMT2019 English-German (En-De) and English-Finnish (En-Fi).
As illustrated in Figure 1, TEL demonstrates superior scaling properties compared to traditional ensembles. While standard ensembles often plateau or even degrade in performance when adding more models (especially beyond 10), TEL's performance remains stable or continues to improve as more models (up to 20) are added.
Table 1 (Figure 6) highlights that TEL achieves state-of-the-art results on WMT2016-2018 tasks. Notably, on Newstest2018 En→De, TEL reached 44.95 BLEU, significantly outperforming both the single baseline (42.42) and the traditional ensemble (44.0). In scenarios where traditional ensembles provided only marginal gains (~0.1-0.3 BLEU), TEL delivered substantial improvements of 0.7-1.0 BLEU.
4. Discussion and Limitations
A key advantage of TEL is its robustness to model diversity. Traditional ensembles are sensitive to "weaker" models, whereas TEL's fine-tuning process allows the model to capture common patterns across translations rather than being forced to average them.
Furthermore, TEL is effective even with limited test data. The authors demonstrated that fine-tuning on as little as 10% of the test set or just the validation set yields significant gains. This suggests the method helps the model adapt to the specific domain or style of the target data. Crucially, after the transductive fine-tuning is complete, only a single model is required for inference, solving the latency issues inherent in traditional ensembles.
The primary limitation of TEL is its requirement that source sentences be known beforehand, making it unsuitable for real-time applications like live chat. Additionally, there is a computational overhead associated with the transductive fine-tuning step for each new document or test set. Future work may explore extending this approach to other NLP tasks such as summarization and text classification.
Summary updated