TL;DR
This paper introduces EvolveMT, a novel machine translation (MT) ensembling system that dynamically selects the best MT engine for each translation request using online learning and a neural quality estimation metric. EvolveMT achieves comparable translation quality to traditional ensembles but with significantly lower computational cost and the unique ability to adapt to changing translation domains without costly retraining.
What Problem Does This Paper Address?
Machine translation (MT) has made significant strides, but effectively combining multiple MT systems (ensembling) remains a challenge. Traditional ensembling methods often rely on training data that may not fully represent future translation requests, leading to a performance mismatch due to data drift. Furthermore, retraining these ensembles is computationally expensive and requires human feedback. This paper tackles the problem of creating an MT ensemble that can adapt to new data and domains in real-time, improving its own performance through usage without the need for expensive retraining or human intervention.
Methods
EvolveMT employs a quasi-ensemble approach, prioritizing the selection of the most suitable MT engine for each segment rather than combining outputs from all. The core of the system is a multi-class classifier that predicts the probability of different MT systems performing best. This classifier is trained using features extracted from the source sentence, including morphological and lexical features, along with embeddings from the XLM-RoBERTa encoder.
A key innovation is the use of a neural quality estimation metric (specifically, COMET-QE) to provide feedback for online learning. When a translation request comes in, EvolveMT extracts features, uses the classifier to predict promising MT engines, and then samples a subset of these engines. The outputs are evaluated using COMET-QE, and the best-performing translation is selected. Crucially, the classifier is then updated online with this information, allowing it to learn from each translation request. This online learning capability enables EvolveMT to adapt dynamically to changes in the translation domain or the MT engines themselves. The system also incorporates an "exploration" mechanism, where it might query additional MT systems if the classifier's predictions are highly uncertain, balancing exploration with exploitation to minimize cost.
Datasets
The paper uses a multi-lingual corpus of 37,500 human-translated sentences from the OPUS repository, covering English-to-Czech, English-to-German, and English-to-Russian language pairs. Translations for evaluation were obtained from one open-source MT system and five major industry MT service providers (Google, Azure, AWS, ModernMT, and AppTek). No new benchmarks were introduced; the focus was on evaluating EvolveMT against existing systems and metrics.
Key Insights and Ideas
- Self-Improving Ensemble: EvolveMT is the first meta MT system that learns and improves its selection strategy during operation, without requiring explicit retraining on human feedback.
- Dynamic Adaptation: The online learning mechanism allows EvolveMT to adapt to data drift and domain changes in real-time, a significant advantage over static ensembles.
- Cost-Quality Trade-off: EvolveMT offers a configurable trade-off between translation quality and computational cost by adjusting the number of MT engines it queries.
- Leveraging Quality Estimation: The integration of a neural quality estimation metric (COMET-QE) as a supervisory signal for online learning is a core enabler of the system's adaptive capabilities.
- Uncertainty-Driven Exploration: The system intelligently explores additional MT engines only when its confidence is low, optimizing resource usage.
Key Results
The experiments demonstrate that EvolveMT achieves translation quality comparable to the COMET-QE ensemble and the "Best MT" baseline, while incurring significantly lower costs. For instance, EvolveMT with MaxMTs = 4 (querying up to 4 engines) provided a good balance between cost and quality across the tested language pairs. The paper also shows that EvolveMT converges quickly, achieving near-optimal performance after only a few hundred translation requests, largely due to the effective use of XLM-ROBERTa embeddings.
Limitations and Future Work
The authors acknowledge that EvolveMT's performance is contingent on the reliability of the COMET-QE model. They also note that batch re-ranking of requests could become a bottleneck if the queue is very large, suggesting asynchronous re-ranking or selecting diverse subsets for re-ranking as potential solutions. Future work could involve caching source embeddings in parallel to speed up feature extraction and optimizing the algorithm to incorporate the cost of each MT engine directly into the ensemble selection process.
Summary updated