LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification

Dheeraj Mekala, Chengyu Dong, Jingbo Shang

Abstract

Weakly supervised text classification methods typically train deep neural classifiers based on pseudo-labels generated from heuristics or seed words. While the quality of these pseudo-labels is crucial to final performance, they are inevitably noisy. Selecting correct samples for training has significant potential for performance gains, yet standard solutions—such as selecting samples based on softmax probability scores—are often ineffective due to the overconfidence of poorly calibrated models. This paper proposes LOPS (Learning Order Inspired Pseudo-Label Selection), a novel method that leverages the "memorization effect" of deep neural networks. By observing that models tend to learn clean patterns before memorizing noisy ones, LOPS selects samples based on their learning order. Extensive experiments across four real-world datasets demonstrate that LOPS serves as a robust, performance-boosting plug-in for existing weakly supervised frameworks.

Introduction

In the absence of massive human-labeled datasets, weakly supervised text classification relies on heuristics—such as matching "seed words" (e.g., labeling a document "Sports" if it contains the word "touchdown")—to generate initial pseudo-labels. These labels are then used to train a classifier, often through iterative processes like self-training or bootstrapping.

However, these heuristics are inherently noisy. If a model is trained on this noisy data without a selection mechanism, it risks a "snowball effect," where the classifier becomes increasingly confident in its own errors. The traditional approach to mitigate this is to select only those samples where the model exhibits high softmax probability. This paper demonstrates that this approach is flawed: modern deep neural networks (DNNs) are often poorly calibrated, assigning high confidence scores (e.g., >0.9) even to incorrectly labeled documents.

The LOPS Methodology

LOPS shifts the selection criteria from how much a model likes a label to when it learns it. This approach is rooted in the "memorization effect," a phenomenon where neural networks prioritize simple, consistent patterns (clean data) in early training stages before eventually brute-forcing the memorization of idiosyncratic, inconsistent patterns (noisy data).

The Selection Process

  1. Probing Phase: A "probing classifier" (such as BERT) is trained on the initial noisy pseudo-labeled dataset.
  2. Tracking Learning Order: For every document, the system records the specific epoch in which the model’s prediction first matches the pseudo-label.
  3. Selection Strategy: Samples that are "learned" in earlier epochs are prioritized, as they are statistically more likely to be correctly labeled.
  4. Thresholding: Training of the probing model is stopped once a predefined percentage ($\tau\%$, typically 50%) of samples for each class have been learned.
  5. Final Training: These high-quality, early-learned samples are then used to train the final production classifier or to seed further rounds of self-training.

Experimental Evaluation

The researchers evaluated LOPS across six diverse datasets, including both coarse-grained and challenging fine-grained versions:

  • NYT-Coarse & NYT-Fine: New York Times news articles.
  • 20News-Coarse & 20News-Fine: The classic 20 Newsgroups dataset.
  • AGNews: A large-scale news categorization dataset.
  • Books: A Goodreads book review dataset categorized by genre.

The "Fine" versions are particularly difficult, containing more classes and higher initial noise ratios (up to 37%).

Key Results

LOPS consistently outperformed standard training and other sophisticated noise-filtering techniques like O2U-Net and MC-Dropout.

  • Performance Gains: On the NYT-Fine dataset using BERT, LOPS improved the Micro-F1 score by 7 points and the Macro-F1 by 10 points over the standard baseline.
  • Model Agnostic: The method proved effective across different architectures, including BERT, XLNet, and GPT-2.
  • Framework Integration: LOPS was successfully integrated into existing weakly supervised frameworks like ConWea and WeSTClass, yielding performance gains across the board.

Discussion and Insights

The Softmax Trap

Experimental visualizations revealed that in the NYT dataset, 60% of wrongly labeled instances still received a BERT confidence score of over 0.9. This confirms that high confidence is an unreliable metric for correctness in noisy environments. LOPS avoids this trap by focusing on training dynamics rather than static probability outputs.

Stability and Robustness

LOPS demonstrates significantly higher stability compared to other selection methods. Methods based on probability or "learning stability" (how often a label remains consistent) showed high variance and occasionally suffered performance crashes on fine-grained datasets. By contrast, the learning order proved to be a consistent and reliable filter for noise.

Conclusion and Future Work

LOPS provides a simple yet powerful "plug-and-play" solution for improving the quality of pseudo-labels in weakly supervised learning. While highly effective, the method does require a sufficiently large pool of initial samples to ensure the selected subset (e.g., the top 50%) is large enough for the model to generalize. Future research may explore increasing the precision of the learning order by measuring at the batch level rather than the epoch level, and expanding the application of the "learning order" principle to image and speech classification tasks.

Summary updated

Abstract

Weakly supervised text classification methods typically train a deep neural classifier based on pseudo-labels. The quality of pseudo-labels is crucial to final performance but they are inevitably noisy due to their heuristic nature, so selecting the correct ones has a huge potential for performance boost. One straightforward solution is to select samples based on the softmax probability scores in the neural classifier corresponding to their pseudo-labels. However, we show through our experiments that such solutions are ineffective and unstable due to the erroneously high-confidence predictions from poorly calibrated models. Recent studies on the memorization effects of deep neural models suggest that these models first memorize training samples with clean labels and then those with noisy labels. Inspired by this observation, we propose a novel pseudo-label selection method LOPS that takes learning order of samples into consideration. We hypothesize that the learning order reflects the probability of wrong annotation in terms of ranking, and therefore, propose to select the samples that are learnt earlier. LOPS can be viewed as a strong performance-boost plug-in to most of existing weakly-supervised text classification methods, as confirmed in extensive experiments on four real-world datasets.
BibTeX
@inproceedings{mekala2022lops,
  author = {Dheeraj Mekala and Chengyu Dong and Jingbo Shang},
  title = {{LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=EMvXsO-52u},
}
Figure