Abstract
Active learning (AL) aims to sample the most informative data instances for labeling, which makes the model fitting data efficient while significantly reducing the annotation cost. However, most existing AL models make a strong assumption that the annotated data instances are always assigned correct labels, which may not hold true in many practical settings. In this paper, we develop a theoretical framework to formally analyze the impact of noisy annotations and show that systematically re-sampling guarantees to reduce the noise rate, which can lead to improved generalization capability. More importantly, the theoretical framework demonstrates the key benefit of conducting active re-sampling on label-efficient learning, which is critical for AL. The theoretical results also suggest essential properties of an active re-sampling function with a fast convergence speed and guaranteed error reduction. This inspires us to design a novel spatial-temporal active re-sampling function by leveraging the important spatial and temporal properties of maximum-margin classifiers. Extensive experiments conducted on both synthetic and real-world data clearly demonstrate the effectiveness of the proposed active re-sampling function.
Introduction
Active Learning (AL) is a "work smarter, not harder" approach to machine learning, where the model selectively queries labels for the most informative data points. However, standard AL algorithms are fragile; if an annotator provides a noisy label for an informative point—often located near the decision boundary—it can catastrophically warp the model's logic. Under moderate noise, standard AL can even perform worse than random sampling. This "Informative Trap" necessitates a strategic approach to re-sampling.
Theoretical Foundation
The authors introduce a theoretical framework analyzing the impact of noisy annotations. A key insight is that while simple re-sampling (e.g., majority voting) eventually reduces noise, active re-sampling—targeting likely errors based on model uncertainty—reduces the noise rate exponentially faster.
The STARS Framework
STARS (Spatial-Temporal Active Re-sampling) addresses noise through three primary pillars:
- Spatial Analysis: It utilizes DEC (Decision function magnitude) to identify points near the decision boundary (Type-I noise) and LIC (Label Inconsistency) to identify outliers (Type-II noise) by comparing a point's label with its neighbors.
- Temporal Scheduling: STARS prioritizes fixing "on-margin" noise early in the training process to stabilize the decision boundary, before moving to address outliers.
- Memorization Unit: To mitigate early-stage learning variance, an exponential moving average tracks re-sampling scores over time, ensuring only consistently suspicious points are flagged.
Experimental Results
STARS was evaluated across synthetic data, real-world tabular datasets (e.g., Yeast, Autodrive), and image datasets (FashionMNIST, CIFAR10).
Key results include:
- Superior Accuracy: On the Autodrive dataset with 30% label noise, STARS reached ~50% accuracy while standard AL reached only ~35%.
- Efficiency: STARS outperformed previous state-of-the-art methods like ImpactEXP while being computationally cheaper.
- Robustness: Even with a limited re-sampling budget, STARS identified critical errors, allowing the model to converge toward clean-data performance limits faster than baselines.
Conclusion
STARS provides a robust framework for label-efficient learning in the presence of noise. By strategically re-sampling points based on spatial and temporal properties, it overcomes the fragility of traditional active learning. Future research will explore non-uniform noise distributions and multi-annotator environments where expertise levels vary.
Summary updated