TL;DR
This paper investigates different strategies for sharing parameters in multi-lingual Neural Machine Translation (NMT) models, finding that task-specific decoder components (especially attention mechanisms) outperform fully shared ones for supervised translation. Crucially, they also introduce and employ novel methods for evaluating zero-shot translation performance across a wide range of language pairs, revealing trade-offs between supervised and zero-shot capabilities.
What Problem Does This Paper Address?
Modern machine translation systems are increasingly aiming for multi-lingual capabilities, allowing a single model to translate between many language pairs. This offers potential efficiency gains and, importantly, the ability to perform "zero-shot" translation – translating between language pairs for which no parallel training data exists. However, it's not always clear how to best design these multi-lingual models. Specifically, the paper addresses the question of how to share parameters within the decoder component of a Transformer-based NMT model across multiple translation tasks. While previous work has explored various sharing strategies, a comprehensive evaluation across a large number of language pairs and translation directions, particularly focusing on zero-shot performance, was lacking. The authors also highlight the practical importance of zero-shot translation, as parallel data is scarce for most language pairs, and it can be a stepping stone for building unsupervised MT systems.
Methods
The core of the paper lies in evaluating different parameter-sharing strategies within the decoder of a Transformer NMT model. They focus on four main configurations:
- PREPEND: A special "task-token" is prepended to the input to signal the target language.
- EMB (Task Embeddings): A unique embedding vector is learned for each target language and combined with the source input.
- DEC (Task-specific Decoders): The entire decoder module is replicated for each target language.
- ATTN (Task-specific Attention): Attention parameters within the decoder are made unique for each target task, while other decoder parameters are shared.
For zero-shot evaluation, they introduce and utilize two novel methods:
- ZERO-SHOT PIVOT: Translating from source to a pivot language, then back to the source, and evaluating the round-trip translation.
- ZERO-SHOT PARALLEL-PIVOT: Translating from source to a pivot language, then to a target language, and evaluating against the target reference. This method is used when parallel data for the target language exists but not for the specific source-target pair.
Datasets
The primary dataset used for training and supervised evaluation is the WMT 2019 shared task parallel datasets. They train on data from 11 distinct languages. For zero-shot evaluation, they leverage the TED Talks multi-parallel corpus and also use the WMT evaluation datasets for their pivot-based evaluations.
Key Insights and Ideas
- Task-specific decoders win for supervised translation: The most surprising finding is that models with task-specific decoder components (especially the ATTN configuration) consistently outperform models with fully shared decoders for supervised translation tasks. This suggests that while sharing offers efficiency, it can come at the cost of performance when sufficient data is available.
- Zero-shot evaluation is tricky: Developing robust methods for evaluating zero-shot translation is crucial. The paper's introduction of the ZERO-SHOT PIVOT and ZERO-SHOT PARALLEL-PIVOT methods provides valuable tools for this.
- Language embeddings can be "fuzzy": The EMB system, which uses language embeddings, showed a higher failure rate in language identification during pivot-based zero-shot evaluation. This suggests that these embeddings might lead to a less distinct representation of target languages, making the model more prone to simply copying input segments.
- Trade-offs are inherent: There's a clear trade-off between maximizing supervised performance and enabling zero-shot capabilities. While task-specific components boost supervised scores, the paper implies that more shared models might be better suited for maximizing zero-shot potential, though this wasn't the primary focus of their evaluation.
Key Results
- For supervised translation, the ATTN (task-specific attention) and DEC (task-specific decoders) models generally outperformed the PREPEND and EMB models across most language pairs. For instance, in Table 4, ATTN and DEC show higher BLEU scores for many pairs compared to PREPEND and EMB.
- The paper conducted a large-scale evaluation covering 110 unique translation directions, with 88 of these being zero-shot directions.
- The EMB system exhibited a significantly higher number of failed pivot tasks due to language identification issues (31 out of 110) compared to other systems (Table 6).
- Overall results (Table 3) show that while supervised scores are generally higher, the zero-shot performance is still substantial, especially for the ATTN and DEC models.
Limitations and Future Work
- Low-resource language pairs: For extremely low-resource language pairs (e.g., EN-GU, EN-KK), the task-specific decoder model (DEC) failed completely, likely due to insufficient updates during training. This highlights a challenge in parameterizing for very low-resource scenarios.
- Copying in zero-shot: The authors acknowledge the risk of models simply copying input segments in zero-shot settings and employ language identification filters to mitigate this. However, they suspect some copying still occurs.
- Future Work: The authors express interest in continuing to study how multi-lingual translation systems scale to realistic volumes of training data and a larger number of source and target tasks. They also aim to explore further improvements in zero-shot performance.
Summary updated