Abstract
To obtain a large amount of training labels inexpensively, researchers have recently adopted the weak supervision (WS) paradigm, which leverages labeling rules to synthesize training labels rather than using individual annotations to achieve competitive results for natural language processing (NLP) tasks. However, data imbalance is often overlooked in applying the WS paradigm, despite being a common issue in a variety of NLP tasks. To address this challenge, we propose Adaptive Ranking-based Sample Selection (ARS2), a model-agnostic framework to alleviate the data imbalance issue in the WS paradigm. Specifically, it calculates a probabilistic margin score based on the output of the current model to measure and rank the cleanliness of each data point. Then, the ranked data are sampled based on both class-wise and rule-aware ranking. In particular, the two sample strategies correspond to our motivations: (1) to train the model with balanced data batches to reduce the data imbalance issue and (2) to exploit the expertise of each labeling rule for collecting clean samples. Experiments on four text classification datasets with four different imbalance ratios show that ARS2 outperformed the state-of-the-art imbalanced learning and WS methods, leading to a 2%-57.8% improvement on their F1-score.
1. Introduction
Deep learning models typically require massive amounts of labeled data to achieve high performance. However, manual annotation is often prohibitively expensive and time-consuming. Weakly Supervised Learning (WSL) has emerged as a cost-effective alternative, allowing researchers to programmatically generate training labels using heuristic rules, patterns, or external knowledge bases.
While WSL effectively scales data labeling, it introduces two significant challenges: label noise and class imbalance. Most existing WSL research focuses on denoising techniques but overlooks the fact that the labeling process itself can amplify existing data imbalances. If a labeling rule for a majority class is more productive than a rule for a minority class, the resulting synthetic dataset becomes even more skewed than the underlying distribution. This paper introduces Adaptive Ranking-based Sample Selection (ARS2), a framework designed to mitigate these issues by ensuring the model learns from a balanced, high-quality subset of data.
2. The ARS2 Framework
ARS2 is a two-stage, model-agnostic framework that identifies "clean" samples and employs strategic ranking to maintain class balance during training.
2.1 Warm-up Stage
Deep neural networks tend to "memorize" simple, clean patterns before they begin to fit to noise. ARS2 leverages this early learning phenomenon by briefly training the model on the full, noisy, weakly-labeled dataset. This initial model acts as a preliminary noise detector for the subsequent selection phase.
2.2 Adaptive Sampling and Ranking
Following the warm-up, the model continues training using only a subset of data selected via three core mechanisms:
- Probabilistic Margin Score (PMS): This is the primary metric for cleanliness. It calculates the difference between the model's confidence in the predicted label and its confidence in the next-best label. A high margin suggests the label is likely clean and reliable.
- Class-wise Ranking (CR): To combat imbalance, ARS2 selects the top-$k$ cleanest samples from each class independently. This prevents majority classes from dominating the training batches and ensures the model receives sufficient signal from minority classes.
- Rule-aware Ranking (RR): This strategy exploits the specific expertise of individual labeling rules. By ranking samples within the scope of each specific rule, the model can better capture the intended signal that the human rule-writer provided.
Additionally, the framework utilizes Dynamic-Rate Sampling, gradually increasing the amount of data the model sees as its discriminative power and confidence improve over time.
3. Experimental Setup
The researchers evaluated ARS2 on four popular text classification benchmarks:
- AGNews: News topic classification (4 classes).
- Yelp: Sentiment analysis (2 classes).
- TREC: Question type classification (6 classes).
- ChemProt: Chemical-protein interaction relation classification (10 classes).
For AGNews and Yelp, the authors created artificially imbalanced versions with ratios as high as 50:1 (where the majority class has 50 times more samples than the minority). TREC and ChemProt were tested in their naturally imbalanced states.
4. Key Results and Insights
ARS2 demonstrated significant performance gains across all tested scenarios:
- F1-Score Improvements: ARS2 outperformed state-of-the-art imbalance learning and WSL methods by 2% to 57.8% in macro-F1 scores.
- Robustness to Extreme Skew: On the Yelp dataset with a 50:1 imbalance ratio, ARS2 maintained high performance while baseline methods (such as Denoise or COSINE) saw their F1 scores plummet.
- Backbone Flexibility: The framework proved effective using both simple Multi-Layer Perceptrons (MLP) and powerful pre-trained models like RoBERTa. When using larger models, the authors implemented a teacher-student approach to prevent the large model from overfitting to noise too quickly.
5. Conclusion and Future Work
ARS2 addresses the critical, often overlooked issue of imbalance amplification in weakly supervised learning. By combining probabilistic margin scores with class-wise and rule-aware ranking, the framework provides a robust solution for training on noisy, skewed datasets. While the current study focuses on text classification, future work may explore extending these ranking strategies to more complex tasks such as sequence labeling (e.g., Named Entity Recognition) or natural language inference.
Summary updated