TL;DR
This paper details the TALP-UPC research group's participation in the WMT18 news translation task for Finnish-English and Estonian-English. They demonstrate that combining training data from these linguistically similar, low-resource languages significantly improves translation quality, especially for Estonian, by leveraging the Transformer architecture and subword vocabulary.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) systems typically require vast amounts of parallel text data to achieve high-quality translations. However, many language pairs, such as Finnish and Estonian, are considered "low-resource," meaning they have limited available training data. This scarcity poses a significant challenge for building effective NMT systems. The authors aim to address this by exploring how to leverage the linguistic similarities between Finnish and Estonian to improve translation quality for these low-resource pairs.
Methods
The TALP-UPC team employed the state-of-the-art Transformer architecture for their NMT systems. This architecture relies solely on attention mechanisms, eschewing recurrent or convolutional networks. A key aspect of their approach was the use of wordpiece vocabulary extraction, a subword tokenization strategy.
Crucially, they investigated the benefit of combining parallel corpora from Finnish-English and Estonian-English. For translation directions where English was the target language (e.g., Finnish to English), they simply concatenated the respective parallel corpora. For directions where English was the source language (e.g., English to Finnish/Estonian), they adopted a strategy inspired by Johnson et al. (2017): they prepended special tokens (e.g., <fi>, <et>) to the English source sentences to indicate the desired target language. This allowed the model to learn language-independent representations and benefit from the combined data.
Datasets
The primary datasets used for training and evaluation were provided by the WMT18 shared task organizers. These included:
- English-Finnish: Europarl corpus (v7 & v8), Paracrawl corpus, Rapid corpus of EU press releases, and Wiki Headlines corpus.
- English-Estonian: Europarl corpus (v8), Paracrawl corpus, and Rapid corpus of EU press releases.
The datasets were preprocessed using standard tools like Moses for normalization, tokenization, and truecasing. A language detection step was also incorporated for the English-Estonian Paracrawl data to filter out noisy, non-Estonian sentences.
Key Insights and Ideas
- Linguistic Similarity is Key: The paper strongly suggests that combining data from linguistically similar languages is a highly effective strategy for low-resource NMT. This allows models to generalize better and learn more robust representations.
- Subword Tokenization Bridges Gaps: Wordpiece tokenization proved crucial in leveraging the shared vocabulary and morphological similarities between Finnish and Estonian, enabling the combined training approach to be successful.
- Target Language Identification is Essential: For English-to-Finnish/Estonian translation, explicitly signaling the target language via prefixes was vital for the model to correctly disambiguate and produce the desired output.
- Estonian Benefits More: The improvements from data combination were more pronounced for Estonian-English translation. This is attributed to the significantly smaller size of the Estonian-English training data compared to Finnish-English, making the relative increase from combining data more impactful.
Key Results
The paper presents a comparison of translation quality (case-insensitive BLEU scores) between systems trained on individual language pairs and those trained on the combined datasets.
- English to Finnish: The combined system achieved a BLEU score of 25.21, a modest improvement of +0.85 over the individual system (24.36).
- Finnish to English: The combined system achieved 30.00 BLEU, a +0.61 improvement over the individual system (29.39).
- English to Estonian: The combined system showed a significant improvement, reaching 18.92 BLEU, a +2.95 increase over the individual system (15.97).
- Estonian to English: The combined system achieved 25.66 BLEU, a substantial +4.00 improvement over the individual system (21.66).
These results clearly indicate that combining data significantly boosts translation quality, particularly for the lower-resource Estonian language pair.
Limitations and Future Work
The authors acknowledge that improvements for Finnish were less dramatic, likely due to its already larger training data size. They suggest that future research should focus on understanding how subword units capture the differences and similarities between cognates in related languages. This insight could lead to even more effective vocabulary extraction strategies for low-resource NMT. The paper also implicitly highlights the need for continued exploration of data augmentation and cross-lingual transfer techniques for low-resource scenarios.
Summary updated