TL;DR
This paper details NECTEC's participation in the WAT-2021 machine translation challenge for the Myanmar-English language pair. They explored multi-source neural machine translation (NMT) models, incorporating syntactic and part-of-speech (POS) information alongside standard text, showing promising improvements for English-to-Myanmar translation.
What Problem Does This Paper Address?
Machine translation (MT) systems have evolved from rule-based approaches to statistical MT (SMT) and now neural MT (NMT). While NMT has achieved state-of-the-art results, existing models often overlook valuable syntactic information. This paper addresses the challenge of improving NMT quality, particularly for the under-resourced Myanmar-English language pair, by investigating how to effectively integrate richer linguistic features like syntactic trees and POS tags into multi-source NMT architectures. This is especially relevant as Myanmar, unlike many languages, lacks clear word segmentation, making linguistic feature extraction more complex.
Methods
The researchers utilized the Marian framework, a C++ based NMT toolkit, to implement their models. They experimented with two primary architectures: the Transformer and an RNN-based encoder-decoder model with an attention mechanism (s2s).
Key methodological contributions include:
- Multi-Source NMT: They implemented multi-source NMT models, which leverage multiple input formats to enhance translation accuracy. This involved combining standard text data with:
- English: Syntactic tree data (using RegexpParser for parsing).
- Myanmar: Part-of-speech (POS) tagged data (using their in-house myWord segmenter and myPOS tagger).
- Model Variants: Within the Transformer architecture, they tested the baseline transformer, multi-transformer (multiple encoders), and shared-multi-transformer (shared parameters between encoders). For the s2s architecture, they tested s2s, multi-s2s, and shared-multi-s2s.
- Data Preprocessing: Crucially, they addressed the lack of word segmentation in Myanmar text by developing and using custom tools for word and syllable segmentation. For English, they explored various parsers (Stanford Parser, BLLIP Parser, Berkeley Neural Parser, RegexpParser) and settled on RegexpParser for its ability to handle long sentences and its simplicity in generating parse trees.
Datasets
The experiments were conducted using parallel corpora provided by the WAT-2021 organizers:
- ALT Corpus: A smaller corpus consisting of twenty thousand Myanmar-English parallel sentences from Wikinews.
- UCSY Corpus: A larger corpus containing 238,014 sentences from various domains, including news articles and textbooks. The Myanmar text in this corpus was not pre-segmented.
Due to hardware limitations, the experiments were primarily conducted using the UCSY corpus for training, with development and test data drawn from the ALT corpus.
Key Insights and Ideas
- Value of Syntactic/POS Information: The integration of syntactic tree data for English and POS tags for Myanmar proved beneficial, particularly for the English-to-Myanmar translation direction. This suggests that richer linguistic features can indeed boost NMT performance, especially for languages with complex morphological structures or segmentation challenges.
- Multi-Source Advantage: The multi-source and shared-multi-source Transformer models showed improvements over the baseline Transformer for English-to-Myanmar translation, highlighting the effectiveness of combining different input modalities.
- Directional Performance Differences: The models performed significantly better for English-to-Myanmar translation than for Myanmar-to-English. This could be attributed to the inherent complexities of the Myanmar language and the challenges in accurately translating from Myanmar to English.
- Preprocessing is Crucial: The paper underscores the critical role of robust data preprocessing, especially for under-resourced languages like Myanmar. Developing custom segmentation and tagging tools was essential for enabling the use of richer linguistic features.
- Error Analysis Reveals Linguistic Gaps: The error analysis identified confusion pairs often stemming from the nature of the Myanmar language (e.g., lack of articles, similar meanings for possessives), incorrect English data cleaning, and fundamental differences in language structures.
Key Results
- English-to-Myanmar:
- The shared-multi-transformer model achieved the highest BLEU score (13.90) and AMFM score (0.654780) among all models.
- Multi-source and shared-multi-source models generally outperformed the baseline Transformer in terms of BLEU and AMFM.
- Myanmar-to-English:
- The baseline models (Transformer and s2s) outperformed the proposed multi-source models in terms of BLEU, RIBES, and AMFM scores. No significant improvements were observed for this direction.
- Word Error Rate (WER):
- For English-to-Myanmar, the baseline Transformer had the lowest WER (81.3%), while for Myanmar-to-English, the baseline s2s had the lowest WER (85.1%). This aligns with the BLEU score observations.
Limitations and Future Work
- Myanmar-to-English Performance: A key limitation is the lack of improvement for the Myanmar-to-English translation task. The authors acknowledge that the proposed models did not surpass the baselines in this direction.
- Hardware Constraints: The researchers faced hardware limitations (2 GPUs, 8GB memory) which prevented them from conducting experiments with both ALT and UCSY training data, forcing them to rely primarily on the UCSY corpus.
- Future Work:
- Post-editing: The authors intend to apply post-editing techniques to improve Myanmar-to-English translation quality.
- String-to-Tree/POS: They plan to extend string-to-tree and string-to-POS translation approaches for under-resourced languages like Myanmar and Thai.
- Addressing Error Causes: Further work could focus on directly addressing the identified causes of confusion pairs, such as improving handling of articles and possessives, and refining data cleaning processes.
Summary updated