Active Learning with Fallible Oracles

254 concepts209 papers4 proposals

Active Learning (AL) is built on a powerful promise: a model can learn more efficiently by choosing which data points it wants labeled. However, traditional AL frameworks rely on the Perfect Oracle Assumption, which posits that the human annotator is an infallible source of truth. In real-world settings—ranging from crowdsourced workers to fatigued domain experts—this assumption rarely holds. When the oracle is fallible, the very mechanism that makes active learning efficient can become its greatest liability.

The Boundary Paradox

The core of the problem lies in what active learning algorithms look for. Most acquisition functions, such as uncertainty sampling, prioritize data points near the model's decision boundary. As Haixia Bi et al. explain in An active semi-supervised deep learning model for human activity recognition, these boundary samples are the most ambiguous and difficult to classify.

This creates a paradox: the samples that provide the most information to the model are exactly the ones where human annotators are most likely to make mistakes. Stanislav Peshterliev et al. note in Active Learning for New Domains in Natural Language Understanding that these boundary cases significantly increase the cognitive load on annotators, leading to higher error rates compared to random sampling. The authors of Evaluating the Impact of Coder Errors on Active Learning demonstrate that while AL is somewhat resilient to random noise, it is highly sensitive to systematic human bias, which can cause it to perform worse than simple random sampling.

The Noise Feedback Loop

When a model receives a noisy label for a sample it is already uncertain about, it enters a destructive feedback loop. Darshan Tank et al. argue in PAL : Pretext-based Active Learning that these incorrect labels corrupt the model’s internal uncertainty metrics. Because the model's "map" of what it doesn't know is now flawed, it begins to select even poorer samples for annotation, leading to a plateau in predictive power.

Hemant Purohit and Ziwei Zhu 0001 highlight in ORIS: Online Active Learning Using Reinforcement Learning-based Inclusive Sampling for Robust Streaming Analytics System that this is often exacerbated by human factors like memory decay and fatigue. If the sampling strategy does not account for the cognitive state of the human, the resulting training data becomes fundamentally unreliable.

Modeling the Human Oracle

To break this loop, researchers have moved toward "Proactive Learning"—a framework that explicitly models the fallibility and cost of different knowledge sources. Yan Yan 0024 et al. proposed the ML+CI* algorithm in Active Learning from Multiple Knowledge Sources to simultaneously identify the most informative data point and the most reliable annotator to query.

Other researchers focus on modeling the specific mechanics of human error:

Robust Selection Strategies

Beyond modeling the human, new selection mechanisms are designed to be inherently robust to noise. Dani Kiyasseh proposes SoQal in SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals, which uses a "Selective Oracle" mechanism. By only bothering a human expert when the model is inconsistent and unconfident, the system filters out potential expert noise by relying on its own high-confidence predictions.

In scenarios where noise is unavoidable, relabeling becomes a critical tool. Christopher H. Lin et al. introduced Re-Active Learning in Re-Active Learning: Active Learning with Relabeling, which uses "Impact Sampling" to decide whether to explore a new data point or relabel an existing one to resolve uncertainty. Similarly, Maolin Li argues in Proactive Learning for Named Entity Recognition that assuming even reliable experts have a non-zero error rate leads to more robust models that are better suited for real-world application.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_perfect_oracle_assumption_label_noise_se_9736fd67,
  title = {perfect oracle assumption (label noise sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perfect-oracle-assumption-label-noise-sensitivity/rdr_9736fd67044e44998e4bc35f73d4085f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_perfect_oracle_assumption_label_noise_se_9736fd67_v1,
  title = {perfect oracle assumption (label noise sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perfect-oracle-assumption-label-noise-sensitivity/rdr_9736fd67044e44998e4bc35f73d4085f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake