STARS: Spatial-Temporal Active Re-sampling for Label-Efficient Learning from Noisy Annotations

Dayou Yu, Weishi Shi, Qi YuAAAI 2023 · OpenReview

Abstract

Active learning (AL) aims to sample the most informative data instances for labeling, which makes the model fitting data efficient while significantly reducing the annotation cost. However, most existing AL models make a strong assumption that the annotated data instances are always assigned correct labels, which may not hold true in many practical settings. In this paper, we develop a theoretical framework to formally analyze the impact of noisy annotations and show that systematically re-sampling guarantees to reduce the noise rate, which can lead to improved generalization capability. More importantly, the theoretical framework demonstrates the key benefit of conducting active re-sampling on label-efficient learning, which is critical for AL. The theoretical results also suggest essential properties of an active re-sampling function with a fast convergence speed and guaranteed error reduction. This inspires us to design a novel spatial-temporal active re-sampling function by leveraging the important spatial and temporal properties of maximum-margin classifiers. Extensive experiments conducted on both synthetic and real-world data clearly demonstrate the effectiveness of the proposed active re-sampling function.

Introduction

Active Learning (AL) is a "work smarter, not harder" approach to machine learning, where the model selectively queries labels for the most informative data points. However, standard AL algorithms are fragile; if an annotator provides a noisy label for an informative point—often located near the decision boundary—it can catastrophically warp the model's logic. Under moderate noise, standard AL can even perform worse than random sampling. This "Informative Trap" necessitates a strategic approach to re-sampling.

Theoretical Foundation

The authors introduce a theoretical framework analyzing the impact of noisy annotations. A key insight is that while simple re-sampling (e.g., majority voting) eventually reduces noise, active re-sampling—targeting likely errors based on model uncertainty—reduces the noise rate exponentially faster.

Theoretical noise reduction: (a) Noise remains rate vs. repeated times R; (b) Noise remains rate vs. initial annotation noise alpha. Active re-sampling (poi) significantly outperforms majority voting (maj).

The STARS Framework

STARS (Spatial-Temporal Active Re-sampling) addresses noise through three primary pillars:

  1. Spatial Analysis: It utilizes DEC (Decision function magnitude) to identify points near the decision boundary (Type-I noise) and LIC (Label Inconsistency) to identify outliers (Type-II noise) by comparing a point's label with its neighbors.
  2. Temporal Scheduling: STARS prioritizes fixing "on-margin" noise early in the training process to stabilize the decision boundary, before moving to address outliers.
  3. Memorization Unit: To mitigate early-stage learning variance, an exponential moving average tracks re-sampling scores over time, ensuring only consistently suspicious points are flagged.

Experimental Results

STARS was evaluated across synthetic data, real-world tabular datasets (e.g., Yeast, Autodrive), and image datasets (FashionMNIST, CIFAR10).

Accuracy vs. Number of Annotated Samples: STARS consistently outperforms "No Re-sample" and "Random Re-sample" baselines, approaching the performance of models trained on clean data (AL_clean).

Key results include:

  • Superior Accuracy: On the Autodrive dataset with 30% label noise, STARS reached ~50% accuracy while standard AL reached only ~35%.
  • Efficiency: STARS outperformed previous state-of-the-art methods like ImpactEXP while being computationally cheaper.
  • Robustness: Even with a limited re-sampling budget, STARS identified critical errors, allowing the model to converge toward clean-data performance limits faster than baselines.

Conclusion

STARS provides a robust framework for label-efficient learning in the presence of noise. By strategically re-sampling points based on spatial and temporal properties, it overcomes the fragility of traditional active learning. Future research will explore non-uniform noise distributions and multi-annotator environments where expertise levels vary.

Summary updated

Abstract

Active learning (AL) aims to sample the most informative data instances for labeling, which makes the model fitting data efficient while significantly reducing the annotation cost. However, most existing AL models make a strong assumption that the annotated data instances are always assigned correct labels, which may not hold true in many practical settings. In this paper, we develop a theoretical framework to formally analyze the impact of noisy annotations and show that systematically re-sampling guarantees to reduce the noise rate, which can lead to improved generalization capability. More importantly, the theoretical framework demonstrates the key benefit of conducting active re-sampling on label-efficient learning, which is critical for AL. The theoretical results also suggest essential properties of an active re-sampling function with a fast convergence speed and guaranteed error reduction. This inspires us to design a novel spatial-temporal active re-sampling function by leveraging the important spatial and temporal properties of maximum-margin classifiers. Extensive experiments conducted on both synthetic and real-world data clearly demonstrate the effectiveness of the proposed active re-sampling function.
BibTeX
@inproceedings{yu2023stars,
  author = {Dayou Yu and Weishi Shi and Qi Yu},
  title = {{STARS: Spatial-Temporal Active Re-sampling for Label-Efficient Learning from Noisy Annotations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=RgR2yFnZE1U},
}
Figure