Selective Sampling for Label Efficient Learning

This direction is merged into another direction: active learning motivation. Its last version stays readable and citable here.

Version 1 of 1115 papers117 conceptsfrozen 2026-10-04

In the standard supervised learning paradigm, we assume that "labels"—the ground-truth answers provided by an expert—are as plentiful as the data itself. But in fields ranging from materials science to autonomous driving, this assumption collapses. Whether it is performing a high-fidelity climate simulation, conducting a clinical trial, or manually segmenting a video frame, every label comes with a hefty price tag in time, money, or human effort.

This research direction focuses on Selective Sampling and Active Learning: the science of identifying which specific data points, if labeled, would most improve a model’s performance. The goal is to achieve the same accuracy as a fully supervised model while querying only a tiny fraction of the available data.

The Version Space and the Power of Disagreement

To understand why selective sampling works, we must first understand the Version Space—the set of all possible models (hypotheses) that are consistent with the labeled data we have seen so far. Early in training, the version space is large; many different models could explain the few examples we have. As we add labels, the version space shrinks.

The most informative data points are those that split the version space most effectively. This is the intuition behind Query By Committee (QBC). As Ran Gilad-Bachrach, Amir Navot, and Naftali Tishby demonstrated in Query by Committee Made Real, if a "committee" of models disagrees on a label, that point lies near the current decision boundary and is highly informative. By projecting high-dimensional version spaces onto lower dimensions, they showed that this theoretical idea could be made computationally efficient for non-linear problems.

This concept of "disagreement" takes many forms. Ion Muslea, Steven Minton, and Craig A. Knoblock explored this through Co-Testing in Active Learning with Strong and Weak Views: A Case Study on Wrapper Induction. They used different "views" (feature sets) of the same data; when these views produced conflicting predictions, the system knew it had found a sample worth labeling. More recently, Aral Hekimoglu and colleagues applied this to autonomous driving in Multi-Task Consistency for Active Learning, identifying informative images by looking for inconsistencies between different tasks, such as object detection and semantic segmentation.

Learning in the Stream

In many real-world applications, data isn't sitting in a static pool; it arrives in a high-velocity stream. In this "online" setting, a model must decide immediately whether to request a label for a passing data point before it is gone forever.

Nicolò Cesa-Bianchi and Claudio Gentile provided a rigorous foundation for this in Linear Classification and Selective Sampling Under Low Noise Conditions, proving that margin-based sampling can converge to an optimal classifier significantly faster than random sampling. This "aggressive" approach—querying labels even when the model is reasonably confident but the margin is small—was further refined by Koby Crammer in Doubly Aggressive Selective Sampling Algorithms for Classification.

The challenge intensifies when the underlying data distribution changes over time, a phenomenon known as concept drift. Bartosz Krawczyk and Alberto Cano 0001 addressed this in Adaptive Ensemble Active Learning for Drifting Data Stream Mining by treating an ensemble of classifiers as a Multi-Armed Bandit, strategically allocating a limited label budget to the most uncertain or drifting parts of the stream.

Beyond Binary Labels

While early work focused on simple classification, the field has expanded to handle more complex "oracles" and data types:

Why It Works: Information Maximization

At its core, selective sampling works because it shifts the objective from "minimizing error on the training set" to "maximizing the information gained per label." Whether using Entropy-Targeted Active Learning to fix biases in materials databases or Information Maximization to find the cleanest statistical signal in a stream, these methods treat the human expert as a precious resource. By focusing on the most informative, ambiguous, or representative samples, selective sampling allows us to build high-performance models in domains where labels were previously too expensive to obtain.

Papers in this version

Showing 60 of 115 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_selective_sampling_for_label_efficient_l_f0de287a,
  title = {Selective Sampling for Label Efficient Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-sampling-for-label-efficient-learning/rdr_f0de287aba5e4601b2b4b8c33996bfc1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_selective_sampling_for_label_efficient_l_f0de287a_v1,
  title = {Selective Sampling for Label Efficient Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-sampling-for-label-efficient-learning/rdr_f0de287aba5e4601b2b4b8c33996bfc1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}