Introduction
Weak supervision (WS) has become a popular paradigm for labeling large-scale datasets using programmatic heuristics or "labeling functions." While WS alleviates the bottleneck of manual annotation, it introduces a significant challenge: class imbalance. Real-world data is often naturally skewed, and the authors of this study demonstrate that the WS process can act as an "imbalance multiplier." If labeling rules for majority classes are more comprehensive or easier to define than those for minority classes, the resulting synthesized dataset becomes even more imbalanced than the original ground truth.
As shown in Figure 1, the imbalance ratio in the TREC dataset increases dramatically from 14 in the ground truth to 1557 in the synthesized labels. Most existing WS methods focus on denoising but fail to address this structural skew, leading to poor performance on critical "tail" classes.
The ARS2 Framework
To address the dual challenges of label noise and class imbalance, the authors propose ARS2 (Adaptive Ranking-based Sample Selection). ARS2 is a model-agnostic framework that employs a two-stage training process:
- Warm-up Stage: The model is initially trained on the full set of noisy synthesized labels. This stage leverages the "memorization effect" of deep neural networks, where clean patterns are learned before the model begins to overfit to noise.
- Continual Training with Adaptive Sampling: Following the warm-up, the model is trained on batches selected through a ranking mechanism designed to identify clean, balanced samples.
The core of ARS2 (illustrated in Figure 2) relies on three key components:
- Probabilistic Margin Score (PMS): Instead of simple confidence, ARS2 uses the margin between the top two predicted class probabilities. A larger margin indicates a higher likelihood that the sample is correctly labeled.
- Class-wise Ranking (CR): To counteract imbalance, the framework selects the top-$k$ cleanest samples per class for each training batch, ensuring minority classes are adequately represented.
- Rule-aware Ranking (RR): This strategy ranks samples based on the reliability of the specific labeling functions that covered them, exploiting the inherent expertise of individual rules.
Experimental Evaluation
The researchers evaluated ARS2 across four benchmarks: AGNews, Yelp, TREC, and ChemProt. For AGNews and Yelp, they simulated varying levels of imbalance, with ratios ranging from 1 (balanced) to 50 (highly imbalanced).
The results demonstrate that ARS2 consistently outperforms state-of-the-art methods in both imbalanced learning (e.g., Logit Adjustment, LDAM) and weak supervision (e.g., COSINE, Denoise). On highly imbalanced datasets, ARS2 achieved F1-macro score improvements ranging from 2% to 57.8%.
Discussion and Conclusion
A key insight from the study is that the Probabilistic Margin Score is a more robust indicator of label quality than standard confidence scores. Furthermore, the authors found that using a simpler "teacher" model (like an MLP) to guide a more complex "student" (like RoBERTa) prevents premature overfitting to noise.
While ARS2 introduces some computational overhead due to periodic dataset ranking, it significantly reduces the need for manual labeling while maintaining high performance on skewed data. Future work may involve extending this adaptive ranking logic to sequence labeling and other complex NLP tasks.
Summary updated