Active Learning from Weak and Strong Labelers

Chicheng Zhang, Kamalika Chaudhuri

Standard active learning assumes access to a single, perfect—but expensive—oracle. This paper introduces a new framework and a provably consistent algorithm that leverages a "weak" (cheap, occasionally incorrect) labeler alongside a "strong" (expensive, expert) one to significantly reduce the number of queries sent to the expert.

What Problem Does This Paper Address?

Active learning (AL) is the art of "learning more with less" by strategically choosing which data points to label. Traditionally, AL assumes you have one source of truth—an "oracle"—that provides perfect labels at a high cost.

However, real-world labeling is rarely that simple. Imagine a medical AI task: you could pay a senior radiologist (the Strong Labeler) a high fee for every image, or you could pay a medical resident (the Weak Labeler) a fraction of that cost. The resident is mostly right but might make mistakes, especially on tricky cases.

The challenge is: How do you use the cheap, weak labeler to save money without "poisoning" your model with incorrect data? Previous theories often assumed the weak labeler’s noise was uniform or followed a simple pattern. This paper tackles a more realistic scenario where the weak labeler can be wrong anywhere and in non-obvious ways.

Methods

The authors build upon Disagreement-Based Active Learning (DBAL). In standard DBAL, the algorithm only requests a label if the current set of "plausible" models disagree on the outcome.

The authors introduce two key innovations to this process to handle the weak labeler:

  1. Cost-Sensitive Difference Classifier: The algorithm learns a secondary model (the "difference classifier") to predict where the Weak and Strong labelers will disagree. Crucially, this classifier is cost-sensitive: it is heavily penalized for "False Negatives" (predicting the labelers agree when they actually don't). This prevents the model from accidentally accepting a wrong label from the weak source, which would lead to statistical bias.
  2. Localized Training: Instead of trying to map out every mistake the weak labeler might make across the entire dataset, the algorithm only trains the difference classifier within the disagreement region of the main task.
Illustration of the optimal classifier's error, the weak labeler's boundary, and the resulting disagreement region between the weak and strong labelers.

As the main model gets smarter and its "uncertainty zone" shrinks, the difference classifier only focuses on that narrowing strip of data. The algorithm proceeds in epochs, refining its understanding of both the target task and the weak labeler’s reliability simultaneously.

Key Insights and Ideas

  • Bias is the Primary Risk: The "aha moment" of the paper is realizing that false negatives in predicting disagreement are much more dangerous than false positives. If you think they disagree but they actually agree, you just wasted a little money querying the expert. If you think they agree but they actually disagree, you've permanently introduced a lie into your training set.
  • The "Agnostic" Advantage: In "agnostic" settings (where no model in your class is perfect), the overhead of learning the difference classifier is actually a "lower-order" concern. This means that in messy, real-world data, the cost of learning when to trust the assistant is negligible compared to the cost of the main learning task.
  • Consistency is Maintained: The authors prove that their algorithm is "statistically consistent," meaning that given enough data, it is guaranteed to find the best possible model, just as if you had only used the expensive expert.

Key Results

This is a theoretical paper focused on establishing mathematical guarantees. Using a case study of Linear Classification under Uniform Distribution (a synthetic setup involving homogeneous linear separators on a $d$-dimensional unit ball), the authors mathematically prove their method achieves better "label complexity" than traditional active learning.

  • Theorem 1 (Consistency): The algorithm is guaranteed to converge to the optimal classifier with high probability, despite using potentially biased labels from the weak source.
  • Theorem 2 (Label Complexity): The authors provide a formal bound on the number of queries to the strong oracle. They show that if the weak labeler agrees with the expert on a significant portion of the "uncertainty zone," the savings are substantial.
  • Asymptotic Gains: In their linear case study, they show that their method improves the label complexity from a factor related to the total disagreement region to a factor related to the much smaller region where the weak labeler is actually wrong.

Limitations and Future Work

  • The "Two-Labeler" Limit: The current framework only considers one weak and one strong labeler. Real-world crowdsourcing often involves dozens of labelers with varying levels of expertise.
  • Computational Overhead: Like many DBAL-style algorithms, this approach requires solving multiple "Empirical Risk Minimization" (ERM) problems with constraints, which can be computationally expensive for complex models like deep neural networks.
  • Random Noise vs. Systematic Bias: The authors note that their algorithm is designed for biased labelers (who are consistently wrong in certain areas). It may perform suboptimally if the weak labeler is just "randomly noisy" everywhere. Future work could aim to unify these two types of noise.

Summary updated

Abstract

An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible.This work addresses active learning with labels obtained from strong and weak labelers, where in addition to the standard active learning setting, we have an extra weak labeler which may occasionally provide incorrect labels. An example is learning to classify medical images where either expensive labels may be obtained from a physician (oracle or strong labeler), or cheaper but occasionally incorrect labels may be obtained from a medical resident (weak labeler). Our goal is to learn a classifier with low error on data labeled by the oracle, while using the weak labeler to reduce the number of label queries made to this labeler. We provide an active learning algorithm for this setting, establish its statistical consistency, and analyze its label complexity to characterize when it can provide label savings over using the strong labeler alone.
BibTeX
@inproceedings{zhang2015active,
  author = {Chicheng Zhang and Kamalika Chaudhuri},
  title = {{Active Learning from Weak and Strong Labelers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2015},
  url = {https://openreview.net/forum?id=H1Mu9_Z_-B},
}
Figure