Abstract
In the field of natural language processing, ensembles are broadly known to be effective in improving performance. This paper analyzes how ensemble of neural machine translation (NMT) models affect performance improvement by designing various experimental setups (i.e., intra-, inter-ensemble, and nonconvergence ensemble). To an in-depth examination, we analyze each ensemble method with respect to several aspects such as different attention models and vocab strategies. Experimental results show that ensembling is not always resulting in performance increases and give noteworthy negative findings.
Introduction
Ensembling—the practice of combining multiple models to improve predictive accuracy—is a cornerstone of competitive Neural Machine Translation (NMT). However, it is often treated as a "black box" performance booster. While ensembling well-fitted models for shared tasks like WMT usually yields gains, there is a significant lack of systematic analysis regarding the conditions under which ensembling fails. This study investigates how vocabulary strategies, attention mechanisms, and architectural heterogeneity (intra- vs. inter-ensemble) influence the final translation quality.
Experimental Setup
The researchers evaluated two primary architectures: LSTM-Attention and Transformer. These were tested across three vocabulary strategies:
- Independent vocab ($ind$): Separate weights for encoder and decoder.
- Share vocab ($sv$): A common vocabulary for both source and target languages.
- Share embedding ($se$): Shared vocabulary plus a shared embedding matrix.
Experiments were conducted on a Korean–English parallel corpus of 1.6 million sentence pairs using SentencePiece tokenization. Performance was measured using BLEU scores across three ensemble types: intra-ensemble (same architecture), inter-ensemble (mixed architectures), and non-convergence ensemble (including under-trained models).
Results and Analysis
Intra-Ensemble Performance
Intra-ensembles generally confirmed the conventional wisdom that combining similar models improves performance. However, the magnitude of these gains was heavily dependent on the vocabulary strategy and model capacity.
As shown in the results, independent vocabularies yielded the highest baseline performance and the most robust ensemble effects. For Transformers with independent vocab, ensembling improved BLEU scores by 0.73 points at a 32k vocabulary size and 1.52 points at 64k. This suggests that broader model capacity is a prerequisite for a successful ensemble effect.
The Inter-Ensemble Trap
A key finding of this research is the "Inter-Ensemble Trap." Contrary to the intuition that architectural diversity should improve generalization, combining LSTMs and Transformers consistently led to performance degradation.
Mixing these heterogeneous architectures resulted in significant BLEU score drops, ranging from -2.02 to -2.94 points compared to the best intra-ensemble baseline. The structural differences between the recurrent nature of LSTMs and the self-attention mechanism of Transformers appear to act as a hindrance rather than a benefit when their predictions are averaged.
Impact of Non-Convergence
The study also explored the impact of "noise" by including under-trained models (trained for only 25% of the required iterations). LSTMs proved highly sensitive to this, with performance dropping by up to 7.05%. Conversely, Transformers showed remarkable resilience. In some specific cases, adding a non-converged model to a Transformer ensemble actually improved results slightly, suggesting that for Transformers, under-fitted models might act as a form of beneficial noise injection that enhances overall robustness.
Conclusion
This research provides a nuanced view of ensembling in NMT. While intra-ensembles are generally effective, especially with independent vocabulary strategies and sufficient model capacity, inter-ensembles between different architectures can be detrimental. These "negative findings" challenge the assumption that diversity always helps and highlight the importance of architectural compatibility in ensemble design. Future work will investigate if these findings generalize across different linguistic typologies beyond the Korean-English pair.
Summary updated