TL;DR
This paper investigates how different subword segmentation strategies impact multilingual machine translation, particularly for low-resource languages. They find that while some methods boost overall model synergy, others are crucial for effective knowledge transfer between languages, with orthographic conventions playing a surprisingly significant role.
What Problem Does This Paper Address?
Multilingual machine translation (MT) models aim to improve translation for low-resource languages by leveraging knowledge from high-resource languages. A key mechanism for this is the use of shared subword units (like "ing" or "un-") across languages, which can create positive transfer (synergy) but also lead to negative interference. The authors highlight that while factors like model size and data proportions have been studied, the specific impact of how text is broken down into subwords (subword segmentation) on this synergy/interference trade-off in multilingual MT remains underexplored. They aim to systematically compare various subword methods to understand their role in promoting synergy and preventing interference, especially in cross-lingual transfer scenarios.
Methods
The study employs two main experimental setups:
- Multilingual Modelling: They train bilingual and trilingual MT models for English translating to four South African languages (Siswati, isiXhosa, Setswana, Afrikaans). They then measure "synergy" and "interference" by comparing the performance of multilingual models against their bilingual counterparts.
- Cross-Lingual Finetuning: They pre-train bilingual MT models and then finetune them on new language pairs. This setup specifically tests how well knowledge transfers from a higher-resource language to a lower-resource one, and how different subword methods facilitate this.
They compare five subword segmentation methods:
- BPE (Byte Pair Encoding): A standard deterministic method that iteratively merges frequent subword pairs.
- ULM (Unigram Language Model): A probabilistic segmenter that can expose models to multiple segmentations for regularization.
- SSMT (Subword Segmental MT): A method that learns subword segmentation jointly with MT training.
- OBPE (Overlap-based BPE): An extension of BPE designed to increase subword overlap across languages in multilingual vocabularies.
- XBPE (eXtended BPE): Extends the vocabulary of a pre-trained model with new subwords for a different translation direction.
Datasets
The experiments utilize the WMT22 dataset for training and the FLORES-101 dataset for validation and testing. For the en→xh/ts/af directions, they sampled sentences to match the size of the en→af data, and removed examples with multi-parallel overlap to isolate the effects of subword segmentation.
Key Insights and Ideas
- ULM for Synergy, BPE for Transfer: ULM consistently leads to greater synergy in multilingual models, improving performance across various linguistic contexts. However, BPE is found to be superior for cross-lingual transfer during finetuning, suggesting a trade-off between methods that promote general model robustness and those that facilitate adaptation to new languages.
- Orthography Matters (A Lot): The study reveals that orthographic word boundary conventions (how words are visually separated) can be a more significant barrier to cross-lingual transfer than linguistic unrelatedness. For instance, the disjunctive orthography of Setswana hinders transfer to Siswati more than the unrelatedness of Afrikaans.
- Subword Regularization Can Hinder Transfer: While ULM's probabilistic segmentation aids multilingual synergy, it can act as a barrier during cross-lingual finetuning. This is because samples from a model trained on one language might not yield suitable subword units when applied to a different language.
Key Results
- Synergy: ULM consistently showed greater synergy across all linguistic contexts compared to other methods.
- Interference: ULM also resulted in minimal interference for higher-resourced languages and even some synergy for en→ts.
- Cross-Lingual Transfer: BPE demonstrated the best cross-lingual transferability during finetuning.
- Linguistic Typology: IsiXhosa provided the most beneficial transfer to Siswati, followed by Setswana and then Afrikaans. However, the orthographic differences between Setswana and Siswati significantly weakened the expected transfer benefits from their relatedness.
Limitations and Future Work
The authors acknowledge that their study is limited to English and four South African languages. They suggest that future work could explore languages with multiple orthographies and conduct more detailed analyses on the interaction between specific subword segmentation methods and language characteristics. They also propose designing multilingual techniques that can look beyond superficial orthographic differences to leverage deeper linguistic similarities.
Summary updated