Introduction
Multilingual Neural Machine Translation (MNMT) has revolutionized the field by enabling a single model to handle dozens of languages. However, these models are typically static. In real-world scenarios, new languages often need to be added after the initial training phase. Traditional approaches to this "incremental learning" problem face a difficult trade-off. Retraining from scratch is computationally prohibitive, while incremental training with a fixed vocabulary leads to high Out-of-Vocabulary (OOV) rates for new languages. Conversely, expanding the vocabulary leads to parameter explosion and representation sparsity, while rebuilding it often triggers "catastrophic forgetting," where the model loses its ability to translate the original languages.
This paper introduces Entropy-Based Vocabulary Substitution (EVS), a novel method designed to add new languages to an existing MNMT model while maintaining a fixed model size and preserving performance on original languages.
The EVS Framework
The core idea of EVS is to intelligently substitute less critical tokens in the original vocabulary with new tokens required for the incremental languages. This process is governed by two primary components: an entropy-based filter and a minimum-cost substitution algorithm.
Entropy-Based Vocabulary Filter
Unlike traditional methods that rely on frequency to determine token importance, EVS utilizes Information Entropy across the multilingual corpus. Tokens with low entropy are considered either universal (appearing consistently across many languages) or highly specific and irreplaceable for a single language. These are retained. In contrast, tokens with high entropy are identified as redundant or "noisy" in a multilingual context and are marked for substitution. This approach ensures that the most semantically vital components of the vocabulary are preserved.
Minimum-Cost Substitution
When a token is replaced, the model must still be able to represent the information it once carried. EVS employs a dynamic programming algorithm to find the "minimum-cost path" to represent discarded tokens using the remaining subwords in the vocabulary. This ensures that the model can still "spell out" removed terms, maintaining sentence length stability and preserving the underlying semantic mappings.
Experimental Evaluation
The researchers tested EVS using a robust setup involving 14 original languages from the WMT-14 benchmark and 7 diverse incremental languages, including Hausa, Tamil, Khmer, Icelandic, Pashto, Japanese, and Polish. This selection covers a wide array of scripts and language families, testing the method's versatility across different linguistic structures.
Key Findings
The experimental results demonstrate that EVS effectively addresses the primary challenges of incremental MNMT.
Superior Translation Quality and Zero OOV
EVS consistently outperformed both "Unadapted" baselines and standard "Adapted" models that rebuild vocabularies. Notably, it achieved a 0% OOV rate for the new languages without increasing the model's parameter count. While expansion-based methods saw their model size grow from 243M to 279M parameters, EVS maintained the original 243M footprint while delivering comparable or superior BLEU scores.
Mitigating Catastrophic Forgetting
A critical advantage of EVS is its ability to preserve performance on the original languages. Traditional adaptation methods often suffer from significant performance drops (up to -21.00 $\Delta$BLEU) on the initial language set. As shown in the performance analysis, the entropy-based approach significantly outperforms frequency-based alternatives in maintaining the integrity of the original translation capabilities.
Operational Efficiency
EVS proved to be highly efficient, achieving a training speed of 1787 tokens/s. This is significantly faster than "Oracle" methods that require full retraining and more efficient than expansion methods, which naturally slow down as the embedding layer grows.
Conclusion and Future Work
Entropy-Based Vocabulary Substitution provides a sustainable, fixed-size solution for the incremental expansion of multilingual translation models. By prioritizing token necessity over simple frequency, it successfully integrates new languages while protecting the model's existing knowledge.
Future research directions include expanding the method to handle non-English-centric language pairs and exploring the use of monolingual data for the incremental step. Additionally, testing the model's stability over multiple successive rounds of updates will be crucial for long-term streaming applications.
Summary updated