TL;DR
This paper tackles the challenging task of building an English-to-Serbian machine translation system for user-generated content, specifically IMDb movie reviews, in a low-resource setting. They found that while neural machine translation (NMT) excels at morphology and syntax, it struggles with lexical aspects like person names, an area where phrase-based MT performs better.
What Problem Does This Paper Address?
Machine translation (MT) systems have made significant strides, with Neural Machine Translation (NMT) largely replacing older Phrase-Based Machine Translation (PBMT) methods. However, NMT systems are particularly sensitive to low-resource scenarios and domain mismatch – situations where the training data (e.g., news articles) differs significantly from the target data (e.g., informal social media text). User-generated content (UGC) like movie reviews presents a unique challenge due to its informal language, noise, and often limited availability of parallel (translated) data. This is further compounded when dealing with morphologically rich languages like Serbian, which also has complex orthographic rules (both Latin and Cyrillic scripts). This paper addresses the specific gap of building an English-to-Serbian MT system for IMDb movie reviews, a low-resource, domain-mismatched scenario, and investigates how to improve performance by generating synthetic in-domain data.
Methods
The authors compare two main MT approaches: PBMT and NMT. They establish baseline systems for both using a clean, out-of-domain news corpus (SEtimes). To address the low-resource and domain mismatch issues, they employ a "forward-translation" technique: they use their baseline systems to translate the monolingual English IMDb reviews into Serbian, creating synthetic in-domain parallel corpora. These synthetic corpora are then used to augment the original SEtimes corpus for training new NMT models. They experiment with different combinations of these corpora to see how they impact translation quality.
Datasets
- SEtimes Corpus: A clean, out-of-domain parallel corpus of news articles used for baseline training.
- IMDb Movie Reviews Dataset: A large dataset of 50,000 English movie reviews, primarily intended for sentiment analysis. For this research, they used 49,800 reviews for training and 200 for testing.
- Synthetic IMDb Corpora: Created by translating the English IMDb reviews into Serbian using their baseline PBMT and NMT systems.
- Serbian Reference Translations: A subset of the test set was human-translated into Serbian for evaluation.
Key Insights and Ideas
- NMT's Strength and Weakness: NMT systems generally outperform PBMT systems, especially in handling the complex morphology and syntax of Serbian. However, NMT struggles significantly with lexical aspects, particularly with person names.
- The Person Name Problem: Translating person names into Slavic languages is a significant challenge. Both PBMT and NMT systems struggled to consistently handle names, generating inconsistent transcriptions, original forms, or even treating them as common nouns. This is a critical area for future research.
- Forward vs. Back-Translation: While synthetic data generation (forward-translation in this case) improved performance, the authors note that "back-translation" (translating natural target language text into the source language) is generally more effective.
- Synthetic Data's Impact: Augmenting the training data with synthetic in-domain parallel corpora improved NMT performance, especially when using data generated by the PBMT system for lexical aspects.
Key Results
- Baseline Performance: The baseline NMT system outperformed the baseline PBMT system on word-level metrics for the test set, despite domain mismatch and low-resource conditions. However, PBMT had better character-level scores.
- Synthetic Data Improvement: Adding synthetic IMDb data (especially from PBMT) improved overall scores, particularly character-level metrics, and reduced lexical errors.
- Error Analysis: NMT systems showed lower inflectional and word order errors compared to PBMT. However, lexical errors were higher for NMT, a finding attributed to the domain mismatch and the challenges with person names.
Limitations and Future Work
- Limited Human Translations: The study was limited by the availability of human-translated Serbian reference data for the test set.
- Forward-Translation Sub-optimality: The "forward-translation" approach used to create synthetic data is acknowledged as less effective than "back-translation."
- Person Name Handling: The inconsistent and often erroneous handling of person names by both MT systems is a major limitation and a key area for future investigation.
- Future Directions: The authors suggest expanding the training data by cleaning subtitle corpora, back-translating monolingual Serbian news articles, and evaluating more IMDb reviews. They also emphasize the need to explore other types of user-generated content (e.g., product reviews, micro-blogs) and other Slavic languages. A systematic investigation into translating person names into Slavic languages is strongly recommended.
Summary updated