LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification

Dheeraj Mekala, Chengyu Dong, Jingbo Shang

Abstract

Weakly supervised text classification relies on "pseudo-labels" generated by heuristics, which are notoriously noisy. While most researchers try to filter these labels using model confidence (softmax probability), this paper introduces LOPS, a method that selects labels based on when a model learns them. By leveraging the "memorization effect"—the tendency of neural networks to learn clean patterns before memorizing noise—LOPS provides a more stable and accurate way to clean up training data.

The Problem: The Overconfidence of Deep Neural Networks

In weakly supervised text classification, we lack massive sets of human-annotated labels. Instead, we use heuristics, such as matching keywords, to create "pseudo-labels." These labels provide a "cold start" for training deep neural networks (DNNs), but they are inevitably messy, often containing 15% to 50% incorrect assignments.

The standard approach to mitigate this is self-training: training a model on pseudo-labels, selecting the ones the model is most "confident" about, and retraining. However, the authors identify a fatal flaw: DNNs are poorly calibrated. They are often "overconfident" in their mistakes. For example, on the NYT dataset, BERT predicts the wrong pseudo-label with over 90% confidence for 60% of the noisy instances. If you rely on probability scores to select data, you end up reinforcing the model's errors—a "snowball effect" of noise.

Figure 3: (a) shows that wrong samples often have high prediction probability (overconfidence), while (b) shows that correct samples are learned significantly earlier than wrong ones.

Methods: Learning Order Inspired Selection

The authors propose LOPS (Learning Order Inspired Pseudo-Label Selection). The core idea is based on the memorization effect: DNNs tend to learn simple, clean patterns in early epochs and only "memorize" noisy, contradictory labels in later epochs.

Figure 4: The LOPS workflow integrated into a self-training pipeline, highlighting the probing classifier and learning-order selection.

The LOPS Workflow:

  1. Probing: Train a "probing classifier" (like BERT) on the noisy pseudo-labeled data.
  2. Tracking: For every document, record the first epoch where the model’s prediction matches the pseudo-label. This is the "Learning Order."
  3. Selection: Instead of taking the most confident samples, take the samples that were "learnt" the earliest.
  4. Thresholding: The algorithm stops training once a certain percentage ($\tau\%$, typically 50%) of samples for each class have been learnt.
  5. Plug-and-Play: These high-quality selected samples are then used to train the final classifier or fed into a standard self-training loop.

By using the timing of learning rather than the magnitude of the final probability, LOPS bypasses the calibration issues that plague modern DNNs.

Experimental Setup and Datasets

The researchers evaluated LOPS across four diverse real-world datasets, including both coarse-grained and fine-grained versions:

  • NYT (New York Times): News articles categorized into 5 coarse genres and 25 fine-grained categories.
  • 20 Newsgroups: A classic dataset partitioned into 6 coarse groups and 20 fine-grained classes.
  • AGNews: A large-scale news collection categorized into 4 topics.
  • Books: A dataset of book descriptions from Goodreads, aiming to predict one of 8 genres.

Key Insights

  • Learning Order > Softmax Probability: The most significant finding is that the order in which a model learns samples is a much more reliable indicator of label correctness than the model's final confidence score.
  • The Calibration Gap: In fine-grained datasets, calibration is so poor that selecting by probability is sometimes worse than picking samples at random. LOPS remains stable even in these difficult scenarios.
  • Invariance to Architecture: Because the memorization effect is a fundamental property of how DNNs optimize, LOPS works consistently across different architectures, including BERT, XLNet, and GPT-2.
  • Strategic Stopping: LOPS provides a natural stopping point for training the probing model, preventing it from reaching the stage where it begins to memorize noise.

Results

LOPS was tested as a "plug-in" for several state-of-the-art weakly supervised frameworks (like ConWea and X-Class).

  • Performance Boost: On the NYT-Fine dataset using BERT, LOPS improved the Micro-F1 score by 7.1 points and the Macro-F1 by 10 points compared to standard training.
  • Stability: Unlike probability-based methods, which showed high variance across different seeds, LOPS demonstrated much lower standard deviation.
  • Efficiency: LOPS effectively curbs noise during the bootstrapping process of self-training, leading to faster and higher convergence in later iterations.

Limitations and Future Work

LOPS requires a relatively high absolute number of pseudo-labeled samples. Because it discards roughly 50% of the data to ensure quality, the final model may struggle to generalize if the initial pool is too small. Future work may explore measuring learning order by "batch number" for finer granularity in large datasets, or applying LOPS to image and speech classification where noisy labels are also a major bottleneck.

Summary updated

Abstract

Weakly supervised text classification methods typically train a deep neural classifier based on pseudo-labels. The quality of pseudo-labels is crucial to final performance but they are inevitably noisy due to their heuristic nature, so selecting the correct ones has a huge potential for performance boost. One straightforward solution is to select samples based on the softmax probability scores in the neural classifier corresponding to their pseudo-labels. However, we show through our experiments that such solutions are ineffective and unstable due to the erroneously high-confidence predictions from poorly calibrated models. Recent studies on the memorization effects of deep neural models suggest that these models first memorize training samples with clean labels and then those with noisy labels. Inspired by this observation, we propose a novel pseudo-label selection method LOPS that takes learning order of samples into consideration. We hypothesize that the learning order reflects the probability of wrong annotation in terms of ranking, and therefore, propose to select the samples that are learnt earlier. LOPS can be viewed as a strong performance-boost plug-in to most of existing weakly-supervised text classification methods, as confirmed in extensive experiments on four real-world datasets.
BibTeX
@inproceedings{mekala2022lops,
  author = {Dheeraj Mekala and Chengyu Dong and Jingbo Shang},
  title = {{LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=EMvXsO-52u},
}
Figure