Abstract
We introduce a new framework for sample-efficient model evaluation that we call active testing. While approaches like active learning reduce the number of labels needed for model training, existing literature largely ignores the cost of labeling test data, typically unrealistically assuming large test sets for model evaluation. This creates a disconnect to real applications, where test labels are important and just as expensive, e.g. for optimizing hyperparameters. Active testing addresses this by carefully selecting the test points to label, ensuring model evaluation is sample-efficient. To this end, we derive theoretically-grounded and intuitive acquisition strategies that are specifically tailored to the goals of active testing, noting these are distinct to those of active learning. As actively selecting labels introduces a bias; we further show how to remove this bias while reducing the variance of the estimator at the same time. Active testing is easy to implement and can be applied to any supervised machine learning method. We demonstrate its effectiveness on models including WideResNets and Gaussian processes on datasets including Fashion-MNIST and CIFAR-100.
Introduction
Machine learning research has extensively explored "Active Learning" to reduce training label costs. However, the cost of labeling test data for model evaluation is often overlooked, with researchers typically assuming the availability of large, pre-labeled test sets. In many real-world applications—such as medical diagnosis or chemical engineering—obtaining labels for evaluation is just as expensive as for training. Standard random (i.i.d.) sampling with a limited budget leads to high-variance estimates, while non-random selection of "hard" cases introduces significant bias. We introduce Active Testing, a framework that strategically selects test points to evaluate models with up to 4x fewer labels than standard methods while remaining mathematically unbiased.
Methods: The LURE Estimator and Acquisition Strategies
Active Testing treats model evaluation as an importance sampling problem. To overcome the bias inherent in non-random selection, we employ the Likelihood-weighted Unbiased Risk Estimator (LURE). This estimator allows us to prioritize informative points—such as those where the model is likely to fail—while weighting their contributions to ensure the final performance estimate is an unbiased reflection of the true test loss.
The framework relies on acquisition functions tailored for evaluation rather than training. For regression, we target points with high predicted error and predictive variance. For classification, we focus on points with high cross-entropy between the model under test and a surrogate model. The surrogate model (e.g., a Gaussian Process or Random Forest) acts as a "scout," predicting where the main model might struggle. The process is iterative: a point is selected via the acquisition function, labeled, and used to update the surrogate.
Experimental Evaluation
We validated Active Testing across diverse settings:
- Synthetic Tasks: Sinusoidal regression and "Two Moons" classification.
- Image Benchmarks: MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.
- Architectures: WideResNets, ResNet-18, Radial Bayesian Neural Networks (BNNs), and Gaussian Processes.
Key Results and Insights
Our experiments demonstrate that Active Testing significantly outperforms naive random sampling. On datasets like CIFAR-10 and Fashion-MNIST, the framework achieved the same precision as random sampling using only 25% of the labels.
A critical insight is that Active Testing is distinct from Active Learning. While active learning often avoids aleatoric uncertainty (noise) to find general trends, active testing seeks out high-noise areas because they contribute significantly to the total loss. Furthermore, we found that surrogate models do not need to be more accurate than the test model; they simply need to have different structural biases to effectively identify the test model's blind spots. Using "Train Ensembles"—models trained once on the training set—proved to be a highly effective and practical surrogate strategy.
Limitations and Future Work
The primary trade-off in Active Testing is the computational overhead of running and potentially updating surrogate models. While effective, the method's efficiency is bounded by the surrogate's ability to identify high-loss regions. Future research could explore more sophisticated acquisition functions and extend the framework to evaluate fairness metrics or safety constraints beyond standard accuracy and loss.
Summary updated