Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

Hongseok Choi, Hyunju LeeIEEE Access 2021 · DOI · OpenReview

TL;DR

This paper tackles the challenge of training effective deep learning models with very limited labeled data (low-resource settings). The authors propose an integrated approach combining a novel weight averaging initialization strategy with a non-validation early stopping method, which consistently outperforms existing techniques and can further boost the performance of state-of-the-art models.

What Problem Does This Paper Address?

In many real-world scenarios, obtaining large amounts of labeled data for machine learning tasks is prohibitively expensive or simply impossible. This "data scarcity" problem is particularly acute in areas like specialized medical text analysis or low-resource languages. While common solutions involve using external data, unlabeled samples, or redesigning model architectures, this paper focuses on a crucial, often overlooked, initial step: how to maximize the utility of the limited labeled data already available without resorting to these more complex methods. The authors specifically target low-resource settings where only 30-100 labeled samples per class are available, aiming to improve model performance before any additional data or architectural changes are considered.

Methods

The core of this research lies in exploring and combining strategies for training deep learning models under severe data constraints. The authors investigate three key aspects:

  • Training-Validation Splitting: They re-examine the efficacy of traditional validation sets in low-resource scenarios, noting that a large validation set can starve the training data, while a small one might lead to ambiguous stopping points.
  • Early Stopping: Beyond standard validation-based early stopping, they propose and evaluate four "non-validation" methods that allow the model to train on all available labeled samples. These include methods like "trn-conf" (stopping based on training sample confidences) and "PE-stop-epoch" (pre-estimated stop epoch).
  • Weight Initialization: They introduce and compare several methods for finding good initial model parameters without architectural changes. Key among these are:
  • Greedy-init: Iteratively trains models on different data splits and selects parameters that perform well on both.
  • WA-init (Weight Averaging Initialization): This is their proposed integrated method. It involves training models on different subsets of data, averaging their parameters, and using this averaged set as the initialization for training on all available samples. This aims to neutralize parameters that are overly specific to certain data subsets while retaining generalizable ones.
  • Meta-init: A data-free method that learns initialization strategies.

The paper then combines the most promising early stopping and initialization methods to propose an "integrated method."

Datasets

The experiments were conducted on six public sentence classification datasets:

  • CR (Customer Review)
  • MR (Movie Reviews)
  • MPQA (Opinion Polarity Classification)
  • TREC (Question Type Classification)
  • SST-2 (Stanford Sentiment Tree - binary classification)
  • SST-5 (Stanford Sentiment Tree - 5-class classification)

For their low-resource setting, they used approximately 2% of the original dataset for training/validation (labeled set $D_l$) and the rest for testing ($D_{tst}$). Specifically, this amounted to 75, 213, 212, 109, 155, and 192 samples per class for CR, MR, MPQA, TREC, SST-2, and SST-5, respectively.

Key Insights and Ideas

  • Large validation sets can be detrimental in low-resource settings: While intuitive for preventing overfitting, using a significant portion of limited data for validation severely restricts the training data, leading to weaker models.
  • Non-validation early stopping methods are crucial: Allowing models to train on all available labeled data, even with a slightly less precise stopping point, can yield better accuracy and loss, especially when combined with good initialization.
  • Weight averaging initialization (WA-init) is a powerful technique: By averaging parameters from models trained on different data subsets, WA-init effectively finds robust initial parameters that generalize better than standard initialization or even parameters from models trained on the entire dataset. This method seems to neutralize overfitting to specific data splits.
  • The "integrated method" (WA-init + trn-conf) is a strong baseline: This combination consistently outperformed other approaches, demonstrating that optimizing training strategy is a fundamental first step in low-resource learning.
  • WA-init is adaptable: The proposed initialization method can be effectively integrated with other advanced techniques like self-training and data augmentation, further boosting their performance.

Key Results

  • The proposed integrated method (WA-init initialization + non-validation early stopping) achieved an average accuracy 1.8% higher than conventional validation-based methods across the six datasets.
  • The WA-init algorithm alone improved average accuracy by 2.0% compared to standard norm-init initialization.
  • In terms of calibration error (ECE), the trn-conf stopping method showed significantly better results than the EB criterion, with ECE and OE being 44% and 53% lower, respectively.
  • When adapted to state-of-the-art models like PERFECT and UST, WA-init consistently provided additional performance improvements, highlighting its broad applicability. For instance, WA-init improved the average accuracy of the PERFECT model by 0.6%.

Limitations and Future Work

  • Parameter Efficiency: The WA-init algorithm requires training multiple models, which can be memory-intensive and time-consuming, especially for very large models. The authors suggest exploring methods that tune only a subset of parameters or leverage pre-trained models more efficiently.
  • Stop Criterion Robustness: The trn-conf stopping criterion relies on accurate confidence estimates. If a model's confidence doesn't well reflect its prediction accuracy (e.g., due to different training/testing processes), trn-conf might not perform optimally. The authors suggest exploring other stop criteria like EB or PE-stop-epoch in such cases.
  • Hyperparameter Tuning: The paper suggests that validation sets might not be adequate for hyperparameter tuning in low-resource settings, as default settings often performed better. Further research could explore more robust hyperparameter optimization strategies for low-resource scenarios.
  • Broader Applicability: The authors express hope that their approach can be applied to real-world low-resource data, such as African languages, and extended to other domains like computer vision. They also aim to study initialization methods that require even fewer parameters.

Summary updated

Abstract

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30–100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training-validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a non-validation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts. Our code is publicly available at https://github.com/DMCB-GIST/exploit_all_samples.
BibTeX
@inproceedings{choi2021exploiting,
  author = {Hongseok Choi and Hyunju Lee},
  title = {{Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2023.3261884},
}
Figure