AUGVIC: Exploiting BiText Vicinity for Low-Resource NMT

Tasnim Mohiuddin, M. Saiful Bari, Shafiq R. Joty

TL;DR

This paper introduces AUGVIC, a novel data augmentation framework for low-resource Neural Machine Translation (NMT) that leverages "vicinal" (nearby) samples within existing parallel data, rather than relying on external monolingual data. AUGVIC significantly improves translation quality, especially when in-domain monolingual data is scarce or mismatched, and can even enhance traditional back-translation methods.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems achieve remarkable performance in high-resource language pairs, but their effectiveness plummets in low-resource settings due to the scarcity of parallel training data. While traditional methods like back-translation (BT) use external monolingual data to generate synthetic parallel data, this approach faces challenges: acquiring large amounts of monolingual data can be expensive, and domain mismatch between the monolingual data and the training/testing bitext can degrade performance. This paper addresses the need for effective NMT augmentation techniques that don't require additional monolingual data and can mitigate domain mismatch issues.

Methods

AUGVIC's core idea is to augment existing parallel data by creating "vicinal" samples around the target sentences. Instead of using external monolingual corpora, it leverages the inherent structure of the bitext. The process involves three main steps:

  1. Vicinal Sample Generation: For each target sentence in the parallel corpus, AUGVIC generates perturbed versions (vicinal samples) using a pretrained masked language model (like XLM-R). This is done without fine-tuning the LM, making the framework flexible. The degree of perturbation is controlled by a "diversity ratio" (ρ), which can be fixed or dynamically adjusted based on sentence length.
  2. Synthetic Bitext Generation: These vicinal target samples are then translated back into the source language to create synthetic parallel pairs. The paper proposes two methods for this:
  3. Pure Back-Translation (PBT): Uses a standard reverse NMT model trained on the original bitext.
  4. Guided Back-Translation (GBT): Employs a Transformer architecture with two encoders. One encoder processes the vicinal target sentence, while the other uses the original source sentence as a "guide" to improve the translation quality of the synthetic source sentence.
  5. Final Model Training: The original parallel data is combined with the newly generated synthetic bitext to train the final source-to-target NMT model. AUGVIC can be used independently or in conjunction with traditional back-translation.

Datasets

The experiments were conducted on four low-resource language pairs: English (En) to/from Bangla (Bn), Tamil (Ta), Nepalese (Ne), and Sinhala (Si). The datasets used were:

  • En-Bn: Islam (2018) - Mixed domains.
  • En-Ta: Ramasamy et al. (2014) - News, Bible, Cinema domains.
  • En-Ne & En-Si: Guzmán et al. (2019) - Bible, GV, PTB, Ubuntu, Open subtitles, Ubuntu domains.

For traditional back-translation experiments, the authors collected and cleaned monolingual data from three domains: News, Wiki, and Gnome.

Key Insights and Ideas

  • Exploiting "Vicinity" is Powerful: The paper demonstrates that the "neighborhood" of existing parallel data is a rich source for augmentation, reducing reliance on external, potentially mismatched, monolingual data.
  • Bridging Domain Gaps: AUGVIC effectively mitigates the negative impact of domain mismatch when combining with external monolingual data, showing higher relative improvements on distant domains.
  • Controlled Diversification: The diversity ratio parameter offers fine-grained control over the augmentation process, allowing researchers to balance diversity and noise.
  • Guided BT for Noisy Data: The Guided Back-Translation (GBT) method shows promise, especially when the original bitext quality is lower, as it uses the original source sentence to guide the synthetic pair generation.

Key Results

  • AUGVIC consistently improves translation quality over the "Bitext baseline" (model trained only on original data), achieving an average gain of +2.76 BLEU across eight translation tasks.
  • It outperforms the "Upsample baseline" (upsampling bitext to match AUGVIC's data size) by an average of 1.75 BLEU.
  • Compared to another diversification method (Nguyen et al., 2020), AUGVIC achieves an average improvement of 0.84 BLEU.
  • When combined with traditional back-translation using external monolingual data, AUGVIC provides additional gains, showing improvements of 1.02 and 0.73 BLEU on average for News and Wiki monolingual data, respectively.
  • The Guided BT approach showed better results on En↔{Bn, Ta} pairs, while Pure BT was better for En↔{Ne, Si}, suggesting that original bitext quality influences the effectiveness of GBT.

Limitations and Future Work

  • Bitext Quality Dependency: The effectiveness of the Guided BT method is shown to be dependent on the quality of the original bitext. Noisy bitexts can propagate errors.
  • Diversity Ratio Tuning: While dynamic diversity ratio generally performs better, finding optimal parameters (a, b, h, tmin, tmax) might require careful tuning.
  • Future Work: The authors suggest that AUGVIC's framework is extendable to utilize stronger LMs as vicinal models in the future. The paper also highlights the potential for using multiple language models for vicinal sample generation to achieve greater diversity. The open-source release of their framework also facilitates further research.

Summary updated

Abstract

The success of Neural Machine Translation (NMT) largely depends on the availability of large bitext training corpora. Due to the lack of such large corpora in low-resource language pairs, NMT systems often exhibit poor performance. Extra relevant monolingual data often helps, but acquiring it could be quite expensive, especially for low-resource languages. Moreover, domain mismatch between bitext (train/test) and monolingual data might degrade the performance. To alleviate such issues, we propose AUGVIC, a novel data augmentation framework for low-resource NMT which exploits the vicinal samples of the given bitext without using any extra monolingual data explicitly. It can diversify the in-domain bitext data with finer level control. Through extensive experiments on four low-resource language pairs comprising data from different domains, we have shown that our method is comparable to the traditional back-translation that uses extra in-domain monolingual data. When we combine the synthetic parallel data generated from AUGVIC with the ones from the extra monolingual data, we achieve further improvements. We show that AUGVIC helps to attenuate the discrepancies between relevant and distant-domain monolingual data in traditional back-translation. To understand the contributions of different components of AUGVIC, we perform an in-depth framework analysis.
BibTeX
@inproceedings{mohiuddin2021augvic,
  author = {Tasnim Mohiuddin and M. Saiful Bari and Shafiq R. Joty},
  title = {{AUGVIC: Exploiting BiText Vicinity for Low-Resource NMT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.267},
}
Figure