Strategic Sample Selection for Label Efficient Learning

This direction is split into other directions: annotation redundancy. Its last version stays readable and citable here.

Version 1 of 1125 papers126 conceptsfrozen 2026-10-04

In the standard machine learning paradigm, we often assume that "more data is better." However, in real-world scenarios—ranging from Named Entity Recognition to molecule generation—human labels are the primary bottleneck. Randomly sampling data for annotation is notoriously inefficient because it treats all data points as equally valuable. In reality, most datasets are riddled with redundant examples that the model already understands, while critical "edge cases" remain unobserved.

The research direction across these papers focuses on Active Learning (AL) and Informed Selection: the art of strategically picking which specific data points a human should label to maximize the model's performance gain per unit of effort.

The Geometry of Uncertainty

The most intuitive way to select data is to find where the model is most "confused." If a model is already 99% certain about a label, asking a human to verify it provides almost zero new information.

Caiming Xiong and colleagues in Active Clustering with Model-Based Uncertainty Reduction formalize this by looking at how a new label would perturb the model's current state. By using matrix perturbation theory, they identify queries that would most drastically reduce the entropy of the system. Similarly, in the context of online model selection, Mohammad Reza Karimi demonstrates that we should only query labels when candidate models in a "pool" disagree. If every model predicts the same thing, the label won't help us distinguish which model is actually the best.

This "uncertainty" isn't just a heuristic; it can be mathematically grounded. For instance, Ojash Neopane uses a "Borda function" in Kernelized Offline Contextual Dueling Bandits to identify which prompts will most effectively teach a Large Language Model (LLM) human preferences with minimal samples.

Beyond Uncertainty: Representativeness and Diversity

Selecting only the "hardest" cases can backfire. If you only label outliers, the model might lose its grasp on the broader distribution of the data. As Saliha Muradoglu notes in Eeny, meeny, miny, moe, while "hard" examples boost performance, "easy" examples can actually degrade accuracy by providing redundant information that overwhelms the learning signal.

To solve this, researchers like Helin Zhao in Active Deep Multi-view Clustering and Ziwei Wang in Deep Hashing with Active Pairwise Supervision argue for a balance of three pillars:

  1. Uncertainty: What the model doesn't know.
  2. Representativeness: How well a sample reflects the underlying density of the unlabeled data.
  3. Diversity: Ensuring the selected samples aren't all clustered in the same region of the feature space.

This is particularly vital in alignment for LLMs. Yuu Jinnai shows that by selecting diverse and representative response texts for human preference labeling, we can align models far more cheaply than by using random pairs.

Adapting to Change and Discovery

Strategic selection is also the key to handling "drift." In social media, language changes rapidly. Shuguang Chen proposes a "trendiness" metric in Mitigating Temporal-Drift to identify which new tweets are most informative for retraining a stale NER model. Instead of labeling everything new, the system targets the samples that represent the shifting distribution.

In some cases, the goal isn't just to improve a known task but to discover new ones. Bingchen Zhao reveals a counter-intuitive finding in Labeled Data Selection for Category Discovery: sometimes, having too much similar labeled data can bias a model, preventing it from discovering new categories. By strategically down-weighting certain samples, they achieve better discovery performance.

Why It Works: The Information Bottleneck

The unifying intuition across these 114 papers is that data is not information. A million redundant data points contain less information than a hundred well-chosen ones.

Whether it's Tuomas Oikarinen using Model-Guided Importance Sampling to evaluate AI neurons 40x more efficiently, or Chiming Duan using active domain adaptation to detect system anomalies with only 2% of the logs, the mechanism is the same: use the model's current state to identify its own blind spots. By turning the model into an active participant in its own training, we move from passive "big data" to "smart data."

Papers in this version

Showing 60 of 125 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_strategic_sample_selection_for_label_eff_48e404a3,
  title = {Strategic Sample Selection for Label Efficient Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/strategic-sample-selection-for-label-efficient-learning/rdr_48e404a3c48045c382510dd2d842b33f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_strategic_sample_selection_for_label_eff_48e404a3_v1,
  title = {Strategic Sample Selection for Label Efficient Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/strategic-sample-selection-for-label-efficient-learning/rdr_48e404a3c48045c382510dd2d842b33f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}