Abstract
Neural Machine Translation (NMT) often fails in low-resource scenarios due to the scarcity of parallel training data and the difficulty of obtaining high-quality in-domain monolingual data for back-translation. We present AUGVIC, a data augmentation framework that generates "vicinal" samples—diverse variations of existing training data—using masked language models. By exploring the neighborhood of the original bitext, AUGVIC achieves significant performance gains without requiring external corpora. Across eight translation tasks, AUGVIC provides an average improvement of +2.76 BLEU points, effectively bridging domain gaps and complementing traditional back-translation techniques.
Introduction
The success of NMT is largely predicated on the availability of massive parallel corpora. For low-resource language pairs, researchers typically rely on Back-Translation (BT), which uses target-side monolingual data to create synthetic training pairs. However, BT faces two critical challenges: the scarcity of high-quality monolingual data in rare languages and the "domain mismatch" problem, where out-of-domain monolingual data introduces noise that can degrade performance.
AUGVIC addresses these issues by leveraging the Vicinal Risk Minimization (VRM) principle. Instead of seeking external data, AUGVIC generates synthetic samples from the "vicinity" of the existing bitext, ensuring the augmented data remains relevant to the target domain.
Methodology
The AUGVIC framework consists of a three-stage pipeline designed to diversify existing bitext:
- Vicinal Sample Generation: Target-side sentences from the bitext are modified using a pretrained masked language model (XLM-R). A Dynamic Diversity Ratio is employed to mask tokens based on sentence length, ensuring that short sentences are sufficiently varied while long sentences maintain semantic coherence.
- Synthetic Bitext Generation: To create source-side counterparts for the vicinal target sentences, the authors propose two methods:
- Pure Back-Translation (PBT): A standard reverse NMT model translates the vicinal sample.
- Guided Back-Translation (GBT): A dual-encoder architecture uses the original source sentence as a "guide" to keep the synthetic translation semantically grounded.
- Final Training: The NMT model is trained on the union of the original bitext and the generated vicinal pairs.
Experimental Setup
The framework was evaluated on four low-resource language pairs: English-Bangla (En-Bn), English-Tamil (En-Ta), English-Nepalese (En-Ne), and English-Sinhala (En-Si). The Ne and Si datasets were specifically chosen to evaluate robustness against domain mismatch, as their training sets (Bible, Ubuntu) differ significantly from their test sets (Wikipedia).
Results and Discussion
AUGVIC demonstrated consistent improvements across all evaluated tasks. As shown in the results table, the method significantly outperforms simple upsampling and other diversification baselines.
Key Findings:
- Additive Benefits: AUGVIC is complementary to traditional BT. When combined with extra monolingual data, it provides further gains, particularly in stabilizing models against distant-domain noise.
- Domain Robustness: In high-mismatch scenarios (En-Ne and En-Si), AUGVIC showed superior stability compared to standard BT, acting as a bridge that prevents the model from drifting toward out-of-domain distributions.
- The Role of the Guide: While GBT is effective for high-quality bitext, PBT proves more robust when the original source data is noisy or poorly segmented, as it avoids propagating existing errors.
Conclusion
AUGVIC offers a powerful, resource-efficient alternative for low-resource NMT. By "squeezing" more value out of existing bitext through vicinal sampling, it mitigates the need for external monolingual data and addresses the domain mismatch problem. Future work will explore scaling this approach with larger generative models and its application to unsupervised translation settings.
Summary updated