Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

Katharina Kann, Kyunghyun Cho, Samuel R. Bowman

Authors: Katharina Kann, Kyunghyun Cho, Samuel R. Bowman
Venue: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP)

Introduction

In standard machine learning workflows, data is partitioned into training, development (validation), and test sets. The development set is indispensable for "early stopping"—the practice of halting training when performance on validation data peaks to prevent overfitting.

However, in low-resource Natural Language Processing (NLP), where data is a scarce commodity, this practice is often applied unrealistically. Researchers frequently simulate low-resource settings by artificially shrinking training sets while maintaining large, high-quality development sets. This creates a "performance signal" that would not exist in real-world applications, potentially leading to research findings that do not generalize to actual low-resource scenarios.

The Development Set Paradox

A review of recent literature reveals a striking trend: in many low-resource studies, the development set is significantly larger than the training set. For example, some setups use only 100 training examples but rely on a 1,000-example development set for model selection and early stopping.

Comparison of training and development set sizes in existing low-resource NLP literature.

In a genuine low-resource environment, a practitioner would likely prioritize the training set, allocating all available data (e.g., all 1,100 examples) to training rather than reserving the majority for validation. The current "status quo" in research thus provides models with an unrealistic advantage.

Methodology: DevSet vs. DevLang

To quantify this effect, the authors compared two training strategies across three sequence-to-sequence tasks: Morphological Inflection (MORPH), Historical Text Normalization (NORM), and Transliteration (TRANSL).

  1. DevSet (The Status Quo): Models are trained using standard early stopping based on a dedicated development set for the specific target language.
  2. DevLang (The Realistic Alternative): This approach assumes no development set is available for the target language. Instead, the optimal number of training epochs is determined by tuning on other related languages (development languages). The model is then trained for that fixed duration on the target language.

Quantifying Performance Inflation

The results demonstrate that relying on a target-language development set generally inflates performance metrics. The impact is most pronounced in tasks highly sensitive to training duration.

Average and maximum performance differences (Δ) between using a target development set (DevSet) and a realistic alternative (DevLang).

In the Morphological Inflection task, the DevSet approach outperformed the realistic DevLang baseline by an average of 1.4% absolute accuracy. However, the variance was high; for certain languages like Azeri, the performance gap reached a staggering 18.0%. This indicates that low-resource models are extremely sensitive to the exact moment training stops; missing the peak by just a few epochs can lead to significant performance degradation.

Interestingly, the effect is task-dependent. In Transliteration, the DevLang approach actually performed 0.5% better on average. This suggests that for some tasks, a small, potentially noisy development set might be less reliable than cross-lingual hyperparameter transfer.

Conclusion and Future Directions

The study concludes that current experimental norms in low-resource NLP often provide an overly optimistic estimate of model performance. To ensure research translates to real-world impact, the authors recommend that the community move away from using large development sets for early stopping in low-resource settings. Instead, researchers should explore more robust methods for determining training duration, such as cross-lingual transfer or other heuristics that reflect true data scarcity.

Summary updated

BibTeX
@inproceedings{kann2019towards,
  author = {Katharina Kann and Kyunghyun Cho and Samuel R. Bowman},
  title = {{Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=nvdzEaRLNZ},
}
Figure