Active Testing: Sample-Efficient Model Evaluation

Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth

Introduction

While "Active Learning" helps us train models with less data, we often forget that labeling test data is just as expensive. In real-world applications—such as medical imaging or experimental physics—acquiring a single label might require hours of an expert's time or expensive lab equipment. This paper introduces Active Testing, a framework that uses clever sampling and bias-correction to evaluate model performance accurately using up to 4x fewer labeled test points than standard random sampling.

The Catch-22 of Test Evaluation

In the ivory tower of machine learning research, we often assume access to massive, perfectly labeled test sets. But if you have a limited budget, you cannot afford to label test points at random (i.i.d. sampling) because many points are "easy" and tell you very little about the model's actual failure modes.

However, if you only pick the "hard" points to label, your evaluation will be biased; your model will look much worse than it actually is. The authors tackle this catch-22: How can we actively pick the most informative test points without breaking the statistical validity of our evaluation?

The Active Testing Framework

The authors propose a framework built on two main pillars:

  1. The LURE Estimator: To fix the "selection bias" introduced by picking specific hard points, they use a technique called importance sampling. They introduce the LURE (Likelihood-Ratio Unbiased Risk Estimator), which weights each labeled point based on how likely it was to be selected. This ensures the final test loss estimate remains mathematically unbiased.
  2. Surrogate-Based Acquisition: To decide which points to label, they use a "surrogate" model (which could be the model under test itself, an ensemble, or even a different architecture). The surrogate estimates the expected loss for unlabeled points. Points with high expected loss are prioritized for labeling because they contribute most to the variance of our estimate.

For regression tasks, the framework prioritizes points where the surrogate is uncertain or disagrees with the main model. For classification, it prioritizes points based on the cross-entropy between the surrogate's and the model's predictions.

Key Insights

  • Active Testing vs. Active Learning: This is a crucial distinction. Active learning often avoids "noisy" areas (aleatoric uncertainty) to find the best decision boundary. Active testing needs to sample those areas because they are exactly where the model is likely to incur loss.
  • The Power of Diversity: A surrogate model doesn't need to be "better" than the model being tested; it just needs to be different. The authors found that using a Random Forest as a surrogate for a Neural Network worked surprisingly well because the two models make different types of mistakes.
  • Retraining Matters: In low-data regimes, retraining the surrogate model every time you get a new test label significantly boosts efficiency. It allows the surrogate to "learn" the error patterns of the main model on the fly.
  • Ensembles are Robust: If you cannot afford to retrain, using a "Train Ensemble" (an ensemble of models trained on the training set) serves as an excellent, stable surrogate for selecting test points.

Empirical Results

The framework was validated across a range of complexities, from synthetic 1D regression to image classification (MNIST, Fashion-MNIST, CIFAR) using models like Gaussian Processes, Bayesian Neural Networks, and ResNets.

Active Testing drastically reduces the variance of the test loss estimate compared to standard I.I.D. sampling.
  • Massive Efficiency Gains: On datasets like Fashion-MNIST and CIFAR-10, Active Testing achieved the same precision as random sampling while using only 25% of the labels (a 4x speedup).
  • Lower Variance: Even with very small budgets (e.g., 5-10 points), the variance of the Active Testing estimator was significantly lower than i.i.d. sampling, leading to much more "trustworthy" early evaluations.
  • Bias Removal: The paper empirically confirms that while naive active selection leads to massive overestimations of error, the LURE estimator successfully recovers the true test loss.
Comparison of different surrogate models on MNIST and Fashion-MNIST. Active Testing with various surrogates consistently outperforms I.I.D. acquisition.

Limitations and Future Work

  • Computational Overhead: Retraining a surrogate model after every single test label acquisition can be computationally expensive, especially for large deep-learning models.
  • Surrogate Quality: If the surrogate is extremely poor or shares the exact same biases as the model under test, the efficiency gains diminish (though the estimate remains unbiased thanks to LURE).
  • Future Directions: The authors suggest exploring more sophisticated surrogates and extending the framework to more complex tasks like object detection or reinforcement learning, where label costs are even higher.

Summary updated

Abstract

We introduce a new framework for sample-efficient model evaluation that we call active testing. While approaches like active learning reduce the number of labels needed for model training, existing...
BibTeX
@inproceedings{kossen2021active,
  author = {Jannik Kossen and Sebastian Farquhar and Yarin Gal and Tom Rainforth},
  title = {{Active Testing: Sample-Efficient Model Evaluation}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=_3ObU8KzeDf},
}
Figure