TL;DR
This paper presents a system for the WMT 2018 biomedical translation task that automatically selects the best translation from a pool of candidates generated by multiple neural machine translation (NMT) models. Their approach focuses on maximizing translation fidelity by using a dictionary and heuristics, offering a simpler alternative to complex ensemble methods.
What Problem Does This Paper Address?
Neural machine translation (NMT) models, while excellent at producing fluent translations, can sometimes struggle with accuracy, particularly with rare words or by omitting parts of the source text. When multiple NMT models are trained, even on similar data, they develop different strengths and weaknesses. The challenge is to effectively combine these diverse models to achieve a translation that is both fluent and highly accurate, without requiring extensive additional training or complex integration.
Methods
The core of the FOKUS system is an "ensemble of translators with automatic selection." Instead of merging outputs or training a meta-model, they generate translations from several NMT models and then automatically select the "best" one. The selection process is based on a heuristic scoring mechanism designed to maximize translation fidelity. This involves:
- Dictionary Matching: Checking how many words in the source sentence have a corresponding translation in the target sentence, using a dictionary.
- Stemming and Normalization: Preprocessing words (both in the source/target sentences and the dictionary) by removing diacritics and applying phonetic substitutions (e.g., 'ph' to 'f') to improve matching.
- Heuristic Rules: Defining rules to score partial matches when exact dictionary entries aren't found. This includes considering word similarity after stemming and normalization, and penalizing short or prefix-based matches.
This method treats the individual NMT models as "black boxes," meaning it doesn't require access to their internal workings or further training beyond what was needed to build the base models.
Datasets
The authors used two main groups of parallel corpora for training and validation:
- MED: UFAL medical corpus 1.0 (English-Romanian).
- NEWS: Times English-Romanian and Europarl 2017 English-Romanian.
They trained models on these datasets in various combinations and evaluated their system on the WMT 2018 biomedical English-to-Romanian translation task.
Key Insights and Ideas
- Simplicity Can Be Effective: The paper demonstrates that a straightforward approach of selecting the best translation from an ensemble, rather than complex fusion techniques, can yield competitive results.
- Fidelity Over Fluency: While NMT excels at fluency, the authors correctly identify fidelity as a key challenge. Their heuristic scoring directly targets this by prioritizing accurate word coverage.
- Diacritics as a Challenge: The Romanian language's complex diacritic system, with historical inconsistencies in character encoding, poses a significant challenge for NLP tasks. Their normalization steps are crucial for handling this heterogeneity.
- Black-Box Ensembling: The ability to use any NMT model as a black box is a significant advantage, allowing for flexibility and integration of diverse translation technologies.
Key Results
The FOKUS system achieved strong results on the WMT 2018 biomedical translation task.
- The best individual model (ID=4) achieved a cased BLEU score of 22.48.
- The ensemble's selected translation achieved a cased BLEU score of 21.73, which was only about 0.5 BLEU points lower than the best individual model and nearly equal to the second-best model (ID=6, BLEU=21.88).
- This indicates that their automatic selection method was highly effective at identifying a top-performing translation from the ensemble.
Limitations and Future Work
- Ensemble Performance Ceiling: The authors acknowledge that their method, by selecting the best from a pre-existing set, cannot outperform the best individual model in the ensemble. It can only reliably select one of the best.
- Dictionary Size: The heuristic scoring's quality is directly tied to the dictionary's size and quality. The dictionary used (approx. 39,000 word pairs) is significantly smaller than comprehensive dictionaries like the Explanatory Dictionary of the Romanian Language (DEX, 65,000 definitions).
- Future Work:
- Using a larger, more comprehensive dictionary.
- Replacing the manual heuristic evaluation with a machine-learned evaluation function trained on parallel corpora.
- Exploring decoder-level combination of word probabilities, rather than just selecting the final output.
Summary updated