Identifying Wrongly Predicted Samples: A Method for Active Learning

Rahaf Aljundi, Nikolay Chumerin, Daniel Olmeda Reino

While unlabeled data is often abundant, the cost of high-quality annotation remains a significant bottleneck in machine learning. Active Learning (AL) addresses this by identifying the most informative samples for labeling. Traditionally, AL has relied on model uncertainty as a proxy for informativeness. However, researchers from Toyota Motor Europe have introduced a novel criterion: Identifying Wrongly Predicted Samples (IWPS). This method shifts the focus from how "confused" a model is to how a sample's current prediction would impact the model's generalization if it were treated as ground truth.

The Core Concept: A "What-If" Analysis

Standard AL methods often fail when a model is "confidently wrong," a common occurrence in imbalanced datasets where rare classes are misclassified as frequent ones. IWPS overcomes this by performing a hypothetical update for every unlabeled sample in the pool.

The process follows three steps:

  1. Hypothesize: Assume the model’s current prediction (pseudo-label) for an unlabeled sample is correct.
  2. Simulate: Perform a small gradient update to the model to reinforce this prediction.
  3. Measure: Evaluate the effect of this update on a small, separate holdout set (validation data).

If the model's prediction was correct, reinforcing it should maintain or improve performance on the holdout set. Conversely, if the prediction was wrong, reinforcing it acts as "poisoning" the model, causing performance on the holdout set to drop. Samples causing the most significant performance degradation are deemed the most informative and prioritized for human annotation.

Figure 1: Illustration of the IWPS mechanism. (a) Initial state. (b) Reinforcing a wrong prediction (green dot) shifts the decision boundary, harming performance on the holdout set (gray dashes). (c) Reinforcing a correct prediction (red dot) has a neutral or positive effect. (d) Final state after selecting the most informative samples.

Scaling with IWPS-app

Running full gradient updates for every sample in a large pool is computationally prohibitive. To address this, the authors derived IWPS-app, a first-order approximation using Taylor expansion. This variant reduces the selection process to a simple dot product between the gradient of the pool sample and the gradient of the holdout set. Mathematically, it selects samples that are most dissimilar to the holdout set in gradient space. This approximation is highly efficient, requiring roughly the time of a single forward pass, making it suitable for large-scale industrial applications.

Experimental Performance

The authors evaluated IWPS across standard benchmarks (MNIST, CIFAR10, Cityscapes) and specifically focused on imbalanced scenarios where some classes were ten times less frequent than others.

Key findings include:

  • Superior Error Detection: On MNIST, IWPS identified wrongly predicted samples at a rate of 90% in the first step, significantly outperforming uncertainty-based methods like BALD (72%).
  • Efficiency in Imbalance: In imbalanced settings, IWPS reached target accuracy levels with half the annotation steps required by random baselines.
  • Minimal Holdout Requirements: The method is surprisingly robust; a holdout set with as few as two samples per class was sufficient to effectively rank the unlabeled pool.
Figure 6: Performance comparison of IWPS and IWPS-app against various baselines (Random, Coreset, BALD, etc.) across different initial training sizes and annotation steps. IWPS consistently maintains a competitive edge, particularly in the early stages of active learning.

Conclusion

IWPS represents a significant departure from uncertainty-centric active learning. By leveraging the impact on generalization error, it provides a more reliable signal for identifying model mistakes, especially in the messy, imbalanced datasets typical of real-world environments. The method is model-agnostic, requires no architectural changes, and offers a computationally efficient approximation that bridges the gap between theoretical robustness and practical utility.

Summary updated

Abstract

While unlabelled data can be largely available and even abundant, the annotation process can be quite expensive and limiting. Under the assumption that some samples are more important for a given task than others, active learning targets the problem of identifying the most informative samples that one should acquire annotations for. In this work we propose a simple sample selection criterion that moves beyond the conventional reliance on model uncertainty as proxy to leverage new labels. By first accepting the model prediction and then judging its effect on the generalization error, we can better identify wrongly predicted samples. We also present a very efficient approximation to our criterion, providing a similarity-based interpretation. In addition to evaluating our method on the standard benchmarks of active learning, we consider the challenging yet realistic imbalanced data scenario. We show state-of-the-art results, especially on the imbalanced setting, and achieve better rates at identifying wrongly predicted samples than existing active learning methods. Our method is simple, model agnostic and relies on the current model status without the need for re-training from scratch.
BibTeX
@inproceedings{aljundi2022identifying,
  author = {Rahaf Aljundi and Nikolay Chumerin and Daniel Olmeda Reino},
  title = {{Identifying Wrongly Predicted Samples: A Method for Active Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=TjBHwYl5Pig},
}
Figure