Critical Gap Between Generalization Error and Empirical Error in Active Learning

Yusuke KanebakoWACV 2024

TL;DR

This paper highlights a critical flaw in how active learning (AL) models are typically evaluated: using cross-validation on AL-selected data leads to an inaccurate estimation of generalization performance. The authors propose a simple yet effective solution: using a fixed, randomly selected subset of data for validation throughout the AL process, which significantly reduces this performance gap.

What Problem Does This Paper Address?

Active learning (AL) aims to reduce annotation costs by intelligently selecting the most informative data points for labeling. A common evaluation practice in AL research involves training a model on a small set of labeled data, then using AL to select more data for labeling and training. Crucially, researchers often evaluate the final model's performance using cross-validation on the data selected by the AL algorithm itself.

The problem is that AL methods, especially those relying on model uncertainty, tend to select data that is biased towards the model's current weaknesses. This means the data selected by AL doesn't accurately represent the true data distribution. Consequently, cross-validation on this biased dataset provides an unreliable estimate of how the model will perform in a real-world, unbiased production environment. This can lead to over-optimistic performance estimates and inefficient data collection.

Methods

The core of the paper's contribution lies in its critique of the standard AL evaluation methodology and its proposed alternative.

  • Critique of Conventional Evaluation: The authors argue that the assumption of having a large, separate annotated test set for evaluation is unrealistic for AL, as AL's goal is to minimize annotation costs. They demonstrate that cross-validation on AL-selected data leads to a significant "gap" between the estimated generalization error and the actual performance. This gap arises because AL methods, particularly uncertainty-based ones, create biased datasets.
  • Proposed Solution: The paper proposes a simple modification to the AL evaluation procedure. Instead of using AL-selected data for validation, they suggest using a fixed set of data that was randomly sampled at the very beginning of the AL process. This initial random set is used only for validation throughout the entire AL cycle, while the AL algorithm continues to select data for training. This approach aims to provide a validation set that better reflects the true data distribution.

Datasets

The experiments were conducted on the following standard image classification datasets:

  • CIFAR-10
  • CIFAR-100
  • SVHN

Key Insights and Ideas

  • The "Gap" is Real and Problematic: The paper convincingly shows that the discrepancy between cross-validation results on AL-selected data and actual generalization performance is not a minor issue but a significant problem that can mislead researchers and practitioners.
  • Uncertainty-Based AL is Inherently Biased: Methods that rely on model uncertainty (e.g., Least Confidence, Entropy, BALD) are shown to select data that is systematically different from the overall data distribution, leading to this evaluation gap.
  • Random Sampling is Less Biased for Evaluation: While random sampling might not be the most efficient for training in AL, it provides a much more reliable dataset for validation because it's less prone to bias.
  • A Simple Fix for a Complex Problem: The proposed solution of using a fixed, randomly selected validation set is surprisingly simple but appears highly effective in bridging the gap between empirical and generalization error. This suggests that the problem is less about the AL algorithms themselves and more about how their performance is measured.

Key Results

  • Significant Gap Observed: Figures 3c, 4, and 6b consistently show a notable gap between test data accuracy and cross-validation accuracy when using AL-selected data for validation, particularly for uncertainty-based AL methods. This gap can be as large as 10-15% in some cases.
  • Proposed Method Reduces the Gap: Figure 7 demonstrates that using a fixed, randomly selected validation set (instead of cross-validation on AL-selected data) significantly suppresses this gap across various AL methods and datasets.
  • Random and Coreset Perform Better in Conventional Evaluation: In the conventional evaluation setting (cross-validation on AL-selected data), Random sampling and Coreset methods showed higher cross-validation accuracy, but this was still an underestimate of true generalization performance.
  • Regularization Helps Training but Not Evaluation Gap: Experiments with Random Augmentation (a form of regularization) improved cross-validation results but did not eliminate the fundamental gap between cross-validation and generalization performance.

Limitations and Future Work

  • Assumption of Uniform Distribution: The paper acknowledges that its proposed solution relies on the assumption that the initial randomly selected data reflects the population distribution. In real-world scenarios, the population itself might be biased or non-uniformly distributed, which could still pose challenges for evaluation.
  • Generalizability to Other Tasks: While tested on image classification, the authors suggest that evaluation methods for other problem settings and tasks need further consideration.
  • Future Work: The authors imply that exploring evaluation strategies for more complex or biased real-world data distributions would be a valuable next step.

Summary updated

BibTeX
@inproceedings{kanebako2024critical,
  author = {Yusuke Kanebako},
  title = {{Critical Gap Between Generalization Error and Empirical Error in Active Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure