TL;DR
This paper introduces AUGVIC, a novel data augmentation framework for low-resource Neural Machine Translation (NMT) that leverages "vicinal" (nearby) samples within existing parallel data, rather than relying on external monolingual data. AUGVIC significantly improves translation quality, especially when in-domain monolingual data is scarce or mismatched, and can even enhance traditional back-translation methods.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) systems achieve remarkable performance in high-resource language pairs, but their effectiveness plummets in low-resource settings due to the scarcity of parallel training data. While traditional methods like back-translation (BT) use external monolingual data to generate synthetic parallel data, this approach faces challenges: acquiring large amounts of monolingual data can be expensive, and domain mismatch between the monolingual data and the training/testing bitext can degrade performance. This paper addresses the need for effective NMT augmentation techniques that don't require additional monolingual data and can mitigate domain mismatch issues.
Methods
AUGVIC's core idea is to augment existing parallel data by creating "vicinal" samples around the target sentences. Instead of using external monolingual corpora, it leverages the inherent structure of the bitext. The process involves three main steps:
- Vicinal Sample Generation: For each target sentence in the parallel corpus, AUGVIC generates perturbed versions (vicinal samples) using a pretrained masked language model (like XLM-R). This is done without fine-tuning the LM, making the framework flexible. The degree of perturbation is controlled by a "diversity ratio" (ρ), which can be fixed or dynamically adjusted based on sentence length.
- Synthetic Bitext Generation: These vicinal target samples are then translated back into the source language to create synthetic parallel pairs. The paper proposes two methods for this:
- Pure Back-Translation (PBT): Uses a standard reverse NMT model trained on the original bitext.
- Guided Back-Translation (GBT): Employs a Transformer architecture with two encoders. One encoder processes the vicinal target sentence, while the other uses the original source sentence as a "guide" to improve the translation quality of the synthetic source sentence.
- Final Model Training: The original parallel data is combined with the newly generated synthetic bitext to train the final source-to-target NMT model. AUGVIC can be used independently or in conjunction with traditional back-translation.
Datasets
The experiments were conducted on four low-resource language pairs: English (En) to/from Bangla (Bn), Tamil (Ta), Nepalese (Ne), and Sinhala (Si). The datasets used were:
- En-Bn: Islam (2018) - Mixed domains.
- En-Ta: Ramasamy et al. (2014) - News, Bible, Cinema domains.
- En-Ne & En-Si: Guzmán et al. (2019) - Bible, GV, PTB, Ubuntu, Open subtitles, Ubuntu domains.
For traditional back-translation experiments, the authors collected and cleaned monolingual data from three domains: News, Wiki, and Gnome.
Key Insights and Ideas
- Exploiting "Vicinity" is Powerful: The paper demonstrates that the "neighborhood" of existing parallel data is a rich source for augmentation, reducing reliance on external, potentially mismatched, monolingual data.
- Bridging Domain Gaps: AUGVIC effectively mitigates the negative impact of domain mismatch when combining with external monolingual data, showing higher relative improvements on distant domains.
- Controlled Diversification: The diversity ratio parameter offers fine-grained control over the augmentation process, allowing researchers to balance diversity and noise.
- Guided BT for Noisy Data: The Guided Back-Translation (GBT) method shows promise, especially when the original bitext quality is lower, as it uses the original source sentence to guide the synthetic pair generation.
Key Results
- AUGVIC consistently improves translation quality over the "Bitext baseline" (model trained only on original data), achieving an average gain of +2.76 BLEU across eight translation tasks.
- It outperforms the "Upsample baseline" (upsampling bitext to match AUGVIC's data size) by an average of 1.75 BLEU.
- Compared to another diversification method (Nguyen et al., 2020), AUGVIC achieves an average improvement of 0.84 BLEU.
- When combined with traditional back-translation using external monolingual data, AUGVIC provides additional gains, showing improvements of 1.02 and 0.73 BLEU on average for News and Wiki monolingual data, respectively.
- The Guided BT approach showed better results on En↔{Bn, Ta} pairs, while Pure BT was better for En↔{Ne, Si}, suggesting that original bitext quality influences the effectiveness of GBT.
Limitations and Future Work
- Bitext Quality Dependency: The effectiveness of the Guided BT method is shown to be dependent on the quality of the original bitext. Noisy bitexts can propagate errors.
- Diversity Ratio Tuning: While dynamic diversity ratio generally performs better, finding optimal parameters (a, b, h, tmin, tmax) might require careful tuning.
- Future Work: The authors suggest that AUGVIC's framework is extendable to utilize stronger LMs as vicinal models in the future. The paper also highlights the potential for using multiple language models for vicinal sample generation to achieve greater diversity. The open-source release of their framework also facilitates further research.
Summary updated