Training on the Test Set: Mapping the System-Problem Space in AI

José Hernández-Orallo, Wout Schellaert, Fernando Martínez-Plumed

Introduction: The Problem with Aggregate Metrics

Current AI evaluation is fundamentally "lossy." When we report a single accuracy percentage for a test set, we discard vital information regarding where and why a system fails. For instance, a delivery robot might boast a 99% success rate overall but fail 100% of the time in specific high-traffic urban centers. These "pockets of failure" are obscured by aggregate metrics.

Furthermore, standard metrics struggle with distribution shift. If an environment changes (e.g., unexpected weather), a static accuracy score becomes obsolete. This paper proposes a shift toward building assessor models: meta-models trained to predict exactly how a specific AI system ($\pi$) will behave on a specific problem instance ($\mu$) before deployment.

The Assessor Model

The authors introduce the Assessor Model, denoted as $\hat{R}(r|\pi, \mu)$, which serves as a conditional probability estimator. Rather than performing the task itself, it predicts the result ($r$) of a system performing a task. This model is trained on historical records of system performance—effectively "training on the test set" to capitalize on evaluation data that is usually discarded.

To be effective, an assessor must meet five key criteria:

The five key traits of a robust assessor model: Anticipative, Standalone, Granular, Behavioural, and Distributional.

Technically, building an assessor involves minimizing empirical error using strictly proper scoring rules like Log-loss or the Brier score, ensuring the model's predictions are well-calibrated probabilities.

Experimental Validation

To demonstrate the concept, the authors utilized the Image Segmentation Data Set, training a population of four different neural networks. The success or failure of these networks on various image attributes (e.g., brickface, foliage, sky) served as the training data for the Assessor Model.

Comparison of actual system accuracy, system self-predicted accuracy, and assessor-predicted accuracy across different image classes.

In these experiments, the assessor model successfully learned to predict the success of specific networks, achieving a Brier Score of 0.086, significantly outperforming a constant baseline (0.124). As shown in the results, the assessor's predictions (green) closely track the actual system accuracy (blue) across various classes, often providing a more reliable estimate than the system's own internal confidence (orange).

Key Insights and Implications

  • Safety and Auditing: Because the assessor is "standalone," it can act as a gatekeeper. If it predicts a high probability of failure for a specific task, the system can decline the action or hand control to a human. Furthermore, it allows for auditing without exposing a company's intellectual property (model weights).
  • Explainability: By varying input features ($\mu$), researchers can interrogate the assessor to find the "tipping point" where a system transitions from success to failure, providing counterfactual explanations for AI behavior.
  • The Meta-OOD Challenge: A primary limitation is that the assessor itself may encounter Out-of-Distribution (OOD) data. If asked to predict performance in a completely novel scenario, its own confidence might be misplaced. Future work must focus on ensuring assessors are perfectly calibrated and possess high-quality features describing both the system and the problem space.

Summary updated

Abstract

Many present and future problems associated with artificial intelligence are not due to its limitations, but to our poor assessment of its behaviour. Our evaluation procedures produce aggregated performance metrics that lack detail and quantified uncertainty about the following question: how will an AI system, with a particular profile \pi, behave for a new problem, characterised by a particular situation \mu? Instead of just aggregating test results, we can use machine learning methods to fully capitalise on this evaluation information. In this paper, we introduce the concept of an assessor model, \hat{R}(r|\pi,\mu), a conditional probability estimator trained on test data. We discuss how these assessors can be built by using information of the full system-problem space and illustrate a broad range of applications that derive from varied inferences and aggregations from \hat{R}. Building good assessor models will change the predictive and explanatory power of AI evaluation and will lead to new research directions for building and using them. We propose accompanying every deployed AI system with its own assessor.
BibTeX
@inproceedings{hernandezorallo2022training,
  author = {José Hernández-Orallo and Wout Schellaert and Fernando Martínez-Plumed},
  title = {{Training on the Test Set: Mapping the System-Problem Space in AI}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21487},
}
Figure