Active Learning via Model Disagreement

371 concepts272 papers4 proposals

Imagine you are trying to find the boundary between two classes in a high-dimensional space. If you randomly sample points from the dataset, you will likely select many instances deep inside the territory of one class, where any reasonable model would already be completely certain. Labeling these points provides no new information and wastes valuable annotation resources.

Instead, suppose you maintain a pool or "committee" of candidate models, all of which are consistent with the data labeled so far but possess slightly different decision boundaries. If you find an unlabeled point where half the committee predicts "Class A" and the other half predicts "Class B," obtaining a true label for this specific point guarantees that you will eliminate a significant portion of the remaining incorrect hypotheses.

This is the core intuition behind disagreement-based active learning: maximizing the information gain per query by targeting the regions of maximum ambiguity among candidate models. As shown by Francesco Strino et al. in VDA, a Method of Choosing a Better Algorithm with Fewer Validations, selecting validation instances based on discordant predictions maximizes the information gained per experimental test. Rather than relying on uniform random sampling, focusing exclusively on data points that can distinguish the relative performance of different models allows practitioners to identify the optimal model much faster. Furthermore, Mengchen Liu demonstrates in Improving Learning-from-Crowds through Expert Validation that selecting data points based on model contradictions is mathematically more effective at identifying misclassified instances than random selection, directly targeting the specific regions of the feature space where the current learning algorithm is most prone to error.

Quantifying Disagreement

To operationalize this intuition, frameworks must mathematically capture the degree of disagreement across unlabeled instances. Several strategies exist depending on the model architecture and task:

  • Variance of Predictions or Losses: In model selection, we can measure the variance of predicted outputs or losses across a set of models. For instance, Satoshi Hara et al. demonstrate in Active model selection: A variance minimization approach that sampling instances where candidate models exhibit the greatest disagreement in predicted losses minimizes the variance of the estimated performance difference. This allows active model selection to find the best model with a fraction of the labels. Similarly, the authors of Online Active Model Selection for Pre-trained Classifiers use the maximum variance of predicted losses among a set of candidate models to guide selective sampling, avoiding wasting labeling budgets on instances where models already agree.
  • Bayesian Disagreement (BALD): When using a single neural network, a committee can be approximated using stochastic realizations of its parameters (e.g., via dropout or Monte Carlo sampling). As Bonaventure F. P. Dossou demonstrates in A Study of Acquisition Functions for Medical Imaging Deep Active Learning, disagreement-based acquisition functions that measure the variance between multiple model inferences consistently outperform standard entropy-based methods by targeting samples that reduce uncertainty in the model's internal parameters. Subhabrata Mukherjee leverages a similar mechanism in Uncertainty-aware Self-training for Few-shot Text Classification, ranking unlabeled data based on the disagreement between different stochastic realizations of the model's parameters to distinguish stable pseudo-labels from areas of high model confusion.
  • Divergence Metrics: For structured outputs like label ranking, Massih Amini et al. show in A Selective Sampling Strategy for Label Ranking that active learning can be implemented by measuring the divergence between a current model and a reference model, targeting instances where the models disagree most on the total ordering of items.
  • PAC-Bayes Priors: In model averaging, Weiran Huang shows in Meta-Learning PAC-Bayes Priors in Model Averaging that prior distributions can be refined more efficiently by sampling data from regions where candidate models show high disagreement, effectively pruning the model space based on predictive volatility.

Generating the Committee

The power of disagreement-based active learning relies heavily on having a diverse committee that accurately represents the remaining uncertainty in the hypothesis space. Researchers have developed several clever ways to construct these committees:

Expanding the Scope: Beyond Simple Classification

While active learning is traditionally framed as finding labels for classification, the principle of model disagreement generalizes across machine learning paradigms:

By focusing on the boundaries where knowledge is most fractured—whether between different algorithms, parameter initializations, or human perspectives—disagreement-based sample selection provides a mathematically rigorous shortcut to sample-efficient learning.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 254 papers
  2. Version 1 · 2026-10-04 · 340 papers
  • co-testing split from this direction (100% overlap)
Cite this direction

Latest version (a living page):

@misc{lacuna_disagreement_based_active_learning_19963364,
  title = {disagreement-based active learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disagreement-based-active-learning/rdr_19963364e35748abac99d9177b963bee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_disagreement_based_active_learning_19963364_v2,
  title = {disagreement-based active learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disagreement-based-active-learning/rdr_19963364e35748abac99d9177b963bee/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake