Abstract
Weakly supervised text classification methods typically train deep neural classifiers based on pseudo-labels generated from heuristics or seed words. While the quality of these pseudo-labels is crucial to final performance, they are inevitably noisy. Selecting correct samples for training has significant potential for performance gains, yet standard solutions—such as selecting samples based on softmax probability scores—are often ineffective due to the overconfidence of poorly calibrated models. This paper proposes LOPS (Learning Order Inspired Pseudo-Label Selection), a novel method that leverages the "memorization effect" of deep neural networks. By observing that models tend to learn clean patterns before memorizing noisy ones, LOPS selects samples based on their learning order. Extensive experiments across four real-world datasets demonstrate that LOPS serves as a robust, performance-boosting plug-in for existing weakly supervised frameworks.
Introduction
In the absence of massive human-labeled datasets, weakly supervised text classification relies on heuristics—such as matching "seed words" (e.g., labeling a document "Sports" if it contains the word "touchdown")—to generate initial pseudo-labels. These labels are then used to train a classifier, often through iterative processes like self-training or bootstrapping.
However, these heuristics are inherently noisy. If a model is trained on this noisy data without a selection mechanism, it risks a "snowball effect," where the classifier becomes increasingly confident in its own errors. The traditional approach to mitigate this is to select only those samples where the model exhibits high softmax probability. This paper demonstrates that this approach is flawed: modern deep neural networks (DNNs) are often poorly calibrated, assigning high confidence scores (e.g., >0.9) even to incorrectly labeled documents.
The LOPS Methodology
LOPS shifts the selection criteria from how much a model likes a label to when it learns it. This approach is rooted in the "memorization effect," a phenomenon where neural networks prioritize simple, consistent patterns (clean data) in early training stages before eventually brute-forcing the memorization of idiosyncratic, inconsistent patterns (noisy data).
The Selection Process
- Probing Phase: A "probing classifier" (such as BERT) is trained on the initial noisy pseudo-labeled dataset.
- Tracking Learning Order: For every document, the system records the specific epoch in which the model’s prediction first matches the pseudo-label.
- Selection Strategy: Samples that are "learned" in earlier epochs are prioritized, as they are statistically more likely to be correctly labeled.
- Thresholding: Training of the probing model is stopped once a predefined percentage ($\tau\%$, typically 50%) of samples for each class have been learned.
- Final Training: These high-quality, early-learned samples are then used to train the final production classifier or to seed further rounds of self-training.
Experimental Evaluation
The researchers evaluated LOPS across six diverse datasets, including both coarse-grained and challenging fine-grained versions:
- NYT-Coarse & NYT-Fine: New York Times news articles.
- 20News-Coarse & 20News-Fine: The classic 20 Newsgroups dataset.
- AGNews: A large-scale news categorization dataset.
- Books: A Goodreads book review dataset categorized by genre.
The "Fine" versions are particularly difficult, containing more classes and higher initial noise ratios (up to 37%).
Key Results
LOPS consistently outperformed standard training and other sophisticated noise-filtering techniques like O2U-Net and MC-Dropout.
- Performance Gains: On the NYT-Fine dataset using BERT, LOPS improved the Micro-F1 score by 7 points and the Macro-F1 by 10 points over the standard baseline.
- Model Agnostic: The method proved effective across different architectures, including BERT, XLNet, and GPT-2.
- Framework Integration: LOPS was successfully integrated into existing weakly supervised frameworks like ConWea and WeSTClass, yielding performance gains across the board.
Discussion and Insights
The Softmax Trap
Experimental visualizations revealed that in the NYT dataset, 60% of wrongly labeled instances still received a BERT confidence score of over 0.9. This confirms that high confidence is an unreliable metric for correctness in noisy environments. LOPS avoids this trap by focusing on training dynamics rather than static probability outputs.
Stability and Robustness
LOPS demonstrates significantly higher stability compared to other selection methods. Methods based on probability or "learning stability" (how often a label remains consistent) showed high variance and occasionally suffered performance crashes on fine-grained datasets. By contrast, the learning order proved to be a consistent and reliable filter for noise.
Conclusion and Future Work
LOPS provides a simple yet powerful "plug-and-play" solution for improving the quality of pseudo-labels in weakly supervised learning. While highly effective, the method does require a sufficiently large pool of initial samples to ensure the selected subset (e.g., the top 50%) is large enough for the model to generalize. Future research may explore increasing the precision of the learning order by measuring at the batch level rather than the epoch level, and expanding the application of the "learning order" principle to image and speech classification tasks.
Summary updated