Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification

Linxin Song, Jieyu Zhang, Tianxiang Yang, Masayuki Goto

Introduction

Weak supervision (WS) has become a popular paradigm for labeling large-scale datasets using programmatic heuristics or "labeling functions." While WS alleviates the bottleneck of manual annotation, it introduces a significant challenge: class imbalance. Real-world data is often naturally skewed, and the authors of this study demonstrate that the WS process can act as an "imbalance multiplier." If labeling rules for majority classes are more comprehensive or easier to define than those for minority classes, the resulting synthesized dataset becomes even more imbalanced than the original ground truth.

Weak supervision amplifies class imbalance in the TREC dataset.

As shown in Figure 1, the imbalance ratio in the TREC dataset increases dramatically from 14 in the ground truth to 1557 in the synthesized labels. Most existing WS methods focus on denoising but fail to address this structural skew, leading to poor performance on critical "tail" classes.

The ARS2 Framework

To address the dual challenges of label noise and class imbalance, the authors propose ARS2 (Adaptive Ranking-based Sample Selection). ARS2 is a model-agnostic framework that employs a two-stage training process:

  1. Warm-up Stage: The model is initially trained on the full set of noisy synthesized labels. This stage leverages the "memorization effect" of deep neural networks, where clean patterns are learned before the model begins to overfit to noise.
  2. Continual Training with Adaptive Sampling: Following the warm-up, the model is trained on batches selected through a ranking mechanism designed to identify clean, balanced samples.
Overview of the ARS2 framework, featuring adaptive sampling through class-wise and rule-aware ranking.

The core of ARS2 (illustrated in Figure 2) relies on three key components:

  • Probabilistic Margin Score (PMS): Instead of simple confidence, ARS2 uses the margin between the top two predicted class probabilities. A larger margin indicates a higher likelihood that the sample is correctly labeled.
  • Class-wise Ranking (CR): To counteract imbalance, the framework selects the top-$k$ cleanest samples per class for each training batch, ensuring minority classes are adequately represented.
  • Rule-aware Ranking (RR): This strategy ranks samples based on the reliability of the specific labeling functions that covered them, exploiting the inherent expertise of individual rules.

Experimental Evaluation

The researchers evaluated ARS2 across four benchmarks: AGNews, Yelp, TREC, and ChemProt. For AGNews and Yelp, they simulated varying levels of imbalance, with ratios ranging from 1 (balanced) to 50 (highly imbalanced).

The results demonstrate that ARS2 consistently outperforms state-of-the-art methods in both imbalanced learning (e.g., Logit Adjustment, LDAM) and weak supervision (e.g., COSINE, Denoise). On highly imbalanced datasets, ARS2 achieved F1-macro score improvements ranging from 2% to 57.8%.

Discussion and Conclusion

A key insight from the study is that the Probabilistic Margin Score is a more robust indicator of label quality than standard confidence scores. Furthermore, the authors found that using a simpler "teacher" model (like an MLP) to guide a more complex "student" (like RoBERTa) prevents premature overfitting to noise.

While ARS2 introduces some computational overhead due to periodic dataset ranking, it significantly reduces the need for manual labeling while maintaining high performance on skewed data. Future work may involve extending this adaptive ranking logic to sequence labeling and other complex NLP tasks.

Summary updated

Abstract

To obtain a large amount of training labels inexpensively, researchers have recently adopted the weak supervision (WS) paradigm, which leverages labeling rules to synthesize training labels rather than using individual annotations to achieve competitive results for natural language processing (NLP) tasks. However, data imbalance is often overlooked in applying the WS paradigm, despite being a common issue in a variety of NLP tasks. To address this challenge, we propose Adaptive Ranking-based Sample Selection (ARS2), a model-agnostic framework to alleviate the data imbalance issue in the WS paradigm. Specifically, it calculates a probabilistic margin score based on the output of the current model to measure and rank the cleanliness of each data point. Then, the ranked data are sampled based on both class-wise and rule-aware ranking. In particular, the two sample strategies corresponds to our motivations: (1) to train the model with balanced data batches to reduce the data imbalance issue and (2) to exploit the expertise of each labeling rule for collecting clean samples. Experiments on four text classification datasets with four different imbalance ratios show that ARS2 outperformed the state-of-the-art imbalanced learning and WS methods, leading to a 2%-57.8% improvement on their F1-score.
BibTeX
@inproceedings{song2022adaptive,
  author = {Linxin Song and Jieyu Zhang and Tianxiang Yang and Masayuki Goto},
  title = {{Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=an9vYun-Ii3},
}
Figure