Abstract
Early stopping, which is widely used to prevent overfitting, is generally based on a separate validation set. However, in low resource settings, validation-based stopping can be risky because a small validation set may not be sufficiently representative, and the reduction in the number of samples by validation split may result in insufficient samples for training. In this study, we propose an early stopping method that uses unlabeled samples. The proposed method is based on confidence and class distribution similarities. To further improve the performance, we present a calibration method to better estimate the class distribution of the unlabeled samples. The proposed method is advantageous because it does not require a separate validation set and provides a better stopping point by using a large unlabeled set. Extensive experiments are conducted on five text classification datasets and several stop-methods are compared. Our results show that the proposed model even performs better than using an additional validation set as well as the existing stop-methods, in both balanced and imbalanced data settings. Our code is available at https://github.com/DMCB-GIST/BUS-stop.
Introduction
Early stopping is a critical regularization technique used to prevent overfitting by halting training when performance on a held-out validation set begins to degrade. However, in low-resource settings—where labeled data is scarce—this approach faces a significant challenge. Splitting a small dataset to create a validation set reduces the already limited training data, often leading to suboptimal models. Furthermore, small validation sets may not be representative of the true data distribution, resulting in noisy stopping signals.
To address this "validation crisis," we propose BUS-stop (Early stopping Based on Unlabeled Samples). This method leverages the abundance of unlabeled data to determine the optimal stopping point without requiring a dedicated validation set.
Methodology
BUS-stop operates in two main stages:
- Preliminary Stage: The model undergoes cross-validation on the small labeled set to establish "reference" behaviors. Specifically, it records the distribution of prediction confidences at the point of peak performance.
- Main Stage: The model is trained on the full labeled dataset. At each epoch, it generates predictions for a large unlabeled set, monitoring two primary metrics:
- Confidence Similarity (conf-sim): This metric compares current prediction confidences on unlabeled data to the reference confidences from Stage 1, acting as a proxy for the loss curve.
- Class Distribution Similarity (class-sim): This measures how closely the predicted class proportions match the estimated true distribution.
As shown in Figure 3, conf-sim effectively captures the long-term trend of the loss, while class-sim is more sensitive to short-term fluctuations in accuracy. By combining these, BUS-stop identifies a fine-grained stopping point. To handle imbalanced data, we incorporate a calibration formula that uses proxy accuracy from the preliminary stage to better estimate the true class distribution of the unlabeled set.
Experimental Results
We evaluated BUS-stop across five text classification benchmarks (SST-2, IMDB, Elec, AG-news, and DBpedia) in low-resource scenarios ranging from 50 to 1,600 samples per class.
The results demonstrate that BUS-stop consistently outperforms standard validation-based stopping (Val-stop_split). Remarkably, it even exceeds the performance of Val-stop_add, where the baseline is provided with additional labeled data for validation. In the most constrained setting (50 samples per class), BUS-stop achieved an average accuracy of 0.872, significantly higher than the 0.826 achieved by standard validation splitting. The method also proved robust in imbalanced data settings, maintaining a lead over other non-validation methods like Evidence-Based (EB) stopping.
Conclusion
BUS-stop offers a powerful alternative to traditional early stopping in low-resource text classification. By utilizing unlabeled data as a "mirror" for the model's internal state, it maximizes the utility of limited labeled samples. While currently optimized for classification, future work will explore its application to generative tasks and integration with self-training techniques.
Code available at: https://github.com/DMCB-GIST/BUS-stop
Summary updated