Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation

Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth

Evaluating a deep learning model is usually straightforward: you take a labeled test set and calculate the average loss. But what if labels are incredibly expensive? Imagine you’re building a model to detect rare diseases in X-rays or classify exotic particles in a supercollider. You might have millions of unlabeled images but only the budget to pay a specialist to label a few hundred.

This is the problem of Active Testing. In a new paper presented at NeurIPS 2022, researchers from the University of Oxford and DeepMind introduce Active Surrogate Estimators (ASEs), a method that fundamentally changes how we approach label-efficient model evaluation.

The Challenge of Active Testing

When labels are scarce, the standard approach is to use Monte Carlo (MC) sampling: pick a few points, get labels, and average the results. To make this more efficient, researchers often use Importance Sampling (IS), which picks "interesting" points (like those where the model is likely to fail) and weights them to keep the estimate unbiased.

However, IS-based methods (like the previous state-of-the-art, LURE) have three major flaws:

  1. Brittleness: If early samples are poor, their weights remain fixed, leading to high variance.
  2. Noise Sensitivity: They struggle when labels are noisy or ambiguous.
  3. Rigidity: They require stochastic sampling with known probabilities to remain unbiased, limiting data selection strategies.

Active Surrogate Estimators (ASE)

The authors propose moving from sampling to interpolation. Instead of just weighting a few labeled points, ASE builds an auxiliary "surrogate" model ($\pi$) to predict the true label distribution for the entire unlabeled pool. The risk estimate is then the average expected loss across the whole pool, calculated using the surrogate's predictions.

ASE is "self-correcting" through Retrospective Correction. Unlike Importance Sampling, where a sample's influence is fixed once it's drawn, ASE updates its predictions for all points in the pool as the surrogate model improves with new data. This approach is also much more robust to label noise because it models the distribution rather than treating labels as absolute ground truth.

The XWED Acquisition Strategy

To decide which points to label, the authors developed Expected Weighted Disagreement (XWED). While standard active learning strategies like BALD look for points where the model is most uncertain, XWED weights that uncertainty by the potential loss. It asks: "Where is the surrogate most confused in a way that actually impacts our final risk estimate?"

Experimental Results

The researchers tested ASEs on standard computer vision benchmarks (MNIST, Fashion-MNIST, CIFAR-10/100), often incorporating distribution shifts to simulate real-world challenges.

Across all datasets, ASEs consistently achieved lower Squared Error than both Naive Monte Carlo and the LURE estimator. Notably, ASE's interpolation approach remained smooth and stable, whereas LURE's estimates often exhibited high-variance "spikes" due to mismatched importance weights.

Squared Error comparison of ASE, LURE, and MC.

In a specific experiment on MNIST where all "7s" were removed from the training set, XWED demonstrated its efficiency in handling distribution shifts. It quickly identified the missing class and focused the labeling budget there, leading to a much faster drop in estimation error compared to other strategies.

Acquisition frequency of the missing '7' class in MNIST.

Conclusion

Active Surrogate Estimators represent a significant step forward in model evaluation under budget constraints. By shifting the focus from re-weighting samples to interpolating with a surrogate model, ASE provides a more robust, stable, and accurate framework for active testing. While retraining the surrogate model adds computational overhead, the gains in label efficiency make it a compelling choice for high-stakes, data-scarce domains.

Summary updated

BibTeX
@inproceedings{kossen2022active,
  author = {Jannik Kossen and Sebastian Farquhar and Yarin Gal and Tom Rainforth},
  title = {{Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=qA81F7XSQf},
}
Figure