Data Cartography for Low-Resource Neural Machine Translation

Aquia Richburg, Marine CarpuatEMNLP 2022 · DOI

Introduction

Neural Machine Translation (NMT) is notoriously data-hungry, yet the vast majority of the world's 7,000 languages lack the massive parallel datasets typically required to train high-quality models. In these "low-resource" settings, every sentence counts. Historically, we haven't understood how a model uses this limited data—whether some sentences are mere noise or the "heavy lifters" for learning. This research applies "Data Cartography" to low-resource NMT to visualize model-data interactions and determine if patterns from simple classification tasks hold true for complex translation.

Methods

The authors adapt Data Cartography (originally by Swayamdipta et al., 2020) for NMT. Every training example is plotted on a 2D coordinate plane based on two metrics calculated during training:

  1. Confidence: The average probability the model assigns to the correct translation across training epochs.
  2. Variability: How much that probability fluctuates during training.

Based on these, data is categorized into three regions: Easy-to-learn (high confidence, low variability), Ambiguous (high variability), and Hard-to-learn (low confidence, low variability).

Data map for Swahili-English (Sw-En) showing the distribution of training samples across Easy, Hard, and Ambiguous regions.

Datasets

The study focuses on two distinct low-resource scenarios:

  • Swahili-English (Sw-En): A "very low-resource" setting with ~63k training samples.
  • Turkish-English (Tr-En): A "low-resource" setting with ~358k samples.

Key Insights and Ideas

Difficulty is Dynamic

A sample's difficulty is not an intrinsic property but a reflection of the model's maturity. A sample that is "hard" during the initial language-modeling phase often becomes "easy" by the time the model focuses on word reordering. The authors decomposed these maps across three training phases: learning grammar (Phase 1), lexical translations (Phase 2), and word reordering (Phase 3).

Data map for Phase 1 of training, illustrating how model confidence and variability evolve early in the learning process.

The "Hard" Data is Not Just Noise

In classification tasks, "hard" samples are often mislabeled or junk data. In low-resource NMT, however, filtering out hard samples often hurts performance. Most samples contribute something to the model's growth. Furthermore, common heuristics like sentence length or word frequency do not strongly correlate with these training dynamics.

Ambiguous Samples are Key

Just as in classification, "ambiguous" samples—those where the model's confidence fluctuates significantly—are among the most helpful for generalization.

Key Results

  • Subset Training: Training on only the "Easy" or "Ambiguous" thirds of the data resulted in significantly better BLEU scores than training on a random third, though it still fell short of using the full dataset.
  • Strategic Filtering: Filtering out the hardest 20% of samples specifically during the final reordering phase (Phase 3) resulted in no loss of quality and, in some cases, a slight improvement (e.g., Sw-En rose from 32.43 to 32.65 BLEU).
  • Predictability: Using a supervised classifier (XLM-RoBERTa), the authors could predict a sample's region with 64-82% accuracy, suggesting latent patterns in data difficulty that simple heuristics miss.

Conclusion

Data cartography reveals that data difficulty in NMT is a moving target. Future work should move toward "model-in-the-loop" data augmentation, generating data specifically designed to move samples from "hard" or "ambiguous" regions into the "easy" region to maximize the efficiency of low-resource training.

Summary updated

BibTeX
@inproceedings{richburg2022data,
  author = {Aquia Richburg and Marine Carpuat},
  title = {{Data Cartography for Low-Resource Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.410},
}
Figure