Active Learning from Weak and Strong Labelers

Chicheng Zhang, Kamalika Chaudhuri

Abstract

An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses active learning with labels obtained from strong and weak labelers, where in addition to the standard active learning setting, we have an extra weak labeler which may occasionally provide incorrect labels. An example is learning to classify medical images where either expensive labels may be obtained from a physician (oracle or strong labeler), or cheaper but occasionally incorrect labels may be obtained from a medical resident (weak labeler). Our goal is to learn a classifier with low error on data labeled by the oracle, while using the weak labeler to reduce the number of label queries made to this labeler. We provide an active learning algorithm for this setting, establish its statistical consistency, and analyze its label complexity to characterize when it can provide label savings over using the strong labeler alone.

Introduction

Standard active learning typically relies on a single, expensive expert (the "strong labeler") to provide ground truth for the most informative data points. However, real-world applications often provide access to "weak labelers"—cheaper sources like crowdsourced workers or junior residents who are less accurate. While using weak labels can reduce costs, a naive integration can introduce unrecoverable bias. If a model incorrectly assumes a weak labeler is accurate in a region where they are not (a false negative in disagreement prediction), the resulting training data becomes permanently biased, preventing convergence to the optimal solution.

This paper introduces a statistically consistent algorithm that safely leverages a weak labeler alongside a strong oracle. By using a cost-sensitive difference classifier and localized training, the approach reduces labeling costs without sacrificing the theoretical guarantees of standard active learning.

Methodology

The proposed framework builds upon Disagreement-Based Active Learning (DBAL) with two critical innovations designed to handle the potential unreliability of the weak labeler.

Cost-Sensitive Difference Classifier

To maintain statistical consistency, the algorithm must avoid false negatives—instances where it wrongly predicts that the weak and strong labelers agree. The authors employ a cost-sensitive learning approach that heavily penalizes these errors. By mathematically constraining the false negative rate to be extremely low, the algorithm ensures that the labels used for training remain virtually unbiased, preserving the integrity of the learned model.

Localized Training and the Disagreement Region

Training a highly accurate difference classifier across the entire input space would be prohibitively expensive in terms of strong labels. Instead, the algorithm localizes its training to the "disagreement region"—the area of uncertainty surrounding the current model's decision boundary. As the algorithm progresses through epochs, this region shrinks, and a fresh difference classifier is trained specifically for this refined space.

Algorithm 4: Testing if a point is in the disagreement region.

The procedure for determining if a point $x$ lies within this disagreement region is detailed in Algorithm 4. It involves comparing the error of a hypothesis trained on the current dataset with one trained on the dataset plus a perturbed label for $x$. If the error difference is below a threshold $\tau$, the point is considered to be in the disagreement region, requiring further scrutiny.

Theoretical Insights and Results

The paper's contributions are primarily theoretical, focusing on consistency and complexity bounds. Unlike empirical papers, this work validates its approach through rigorous mathematical proofs and a theoretical case study using a uniform distribution over a $d$-dimensional unit ball.

  • Provable Consistency: The authors prove that their algorithm is statistically consistent (Theorem 1). Its error reliably converges to the error of the best possible classifier in the hypothesis class, despite the use of potentially noisy weak labels.
  • Label Complexity: Theorem 2 establishes a rigorous bound on the number of strong oracle queries. The analysis shows that the overhead of training the difference classifier is relatively small. In agnostic settings (where perfect classification is impossible), this overhead becomes negligible compared to the labels required for the primary task.
  • Asymptotic Improvements: A theoretical case study on linear classification demonstrates that if the weak labeler's errors are concentrated near the decision boundary but accurate elsewhere, the algorithm achieves strictly better asymptotic label complexity than standard DBAL using only the strong oracle.

Limitations and Future Work

While robust against systematic bias, the algorithm is currently sub-optimal for handling purely random, uniform noise from a weak labeler. Additionally, the current framework is limited to a single weak labeler. Future research directions include extending the model to multiple weak labelers with varying areas of expertise and developing methods that simultaneously address both systematic bias and random noise in weak annotators.

Summary updated

Abstract

An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses active learning with labels obtained from strong and weak labelers, where in addition to the standard active learning setting, we have an extra weak labeler which may occasionally provide incorrect labels. An example is learning to classify medical images where either expensive labels may be obtained from a physician (oracle or strong labeler), or cheaper but occasionally incorrect labels may be obtained from a medical resident (weak labeler). Our goal is to learn a classifier with low error on data labeled by the oracle, while using the weak labeler to reduce the number of label queries made to this labeler. We provide an active learning algorithm for this setting, establish its statistical consistency, and analyze its label complexity to characterize when it can provide label savings over using the strong labeler alone.
BibTeX
@inproceedings{zhang2015active,
  author = {Chicheng Zhang and Kamalika Chaudhuri},
  title = {{Active Learning from Weak and Strong Labelers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=H1Mu9_Z_-B},
}
Figure