Interpretable Meta-Score for Model Performance: Extended Abstract

Alicja Gosiewska, Katarzyna Woznica, Przemyslaw BiecekMeta-Knowledge Transfer @ ECML/PKDD 2022 · OpenReview

Abstract

As machine learning benchmarks proliferate across domains like computer vision and natural language processing, the community faces a growing challenge: how to definitively determine if one model outperforms another. Traditional metrics such as AUC, accuracy, or RMSE provide numerical assessments of specific properties but fail to offer a probabilistic understanding of relative performance, statistical significance, or comparability across disparate datasets. To address these shortcomings, we introduce EPP (Estimated Performance Probability), a meta-measure inspired by the Elo ranking system used in competitive games. By framing models as "players" and performance metrics as "scores" within a Unified Benchmark Ontology, EPP provides a universal, interpretable, and statistically grounded framework for model evaluation.

The Limitations of Traditional Benchmarking

While standard metrics tell us how well a model performed on a specific task, they possess three significant "blind spots" that hinder comparative analysis:

  1. Lack of Probabilistic Insight: Traditional metrics do not quantify the probability that Model A will outperform Model B in a future scenario.
  2. Ambiguous Statistical Significance: It is often unclear whether a marginal improvement (e.g., a 0.01 increase in AUC) represents a true performance lead or mere statistical noise.
  3. Dataset Incomparability: Metrics are highly sensitive to dataset difficulty. An AUC improvement of 0.01 on a nearly "solved" dataset is significantly harder to achieve than the same improvement on a difficult one, yet standard averaging treats these gains as equivalent.

The EPP Framework

The EPP meta-score addresses these issues by borrowing the logic of the Elo rating system. To ensure the framework is universally applicable, we propose the Unified Benchmark Ontology:

  • Players: The models or architectures being evaluated.
  • Rounds: The specific evaluation context, such as a dataset or a cross-validation fold.
  • Score: The raw performance metric (AUC, Accuracy, etc.) obtained in a round.

Mathematical Foundation

EPP estimates the probability ($p_{i,j}$) that Model $i$ will outperform Model $j$ using a logistic regression framework. The log-odds of winning are determined by the difference between the models' meta-scores ($\beta$):

$$\log\left(\frac{p_{i,j}}{1 - p_{i,j}}\right) = \beta_{M_i} - \beta_{M_j}$$

By focusing on the frequency of "wins" across multiple rounds rather than simple metric averages, EPP aggregates results across diverse datasets and metrics into a single, unified ranking.

Key Insights and Advantages

Stability and Consistency

Unlike traditional averaging, EPP treats stability as a primary factor. A model that performs exceptionally well in one round but fails in another will receive a lower EPP score and higher uncertainty than a consistent, mid-tier performer. This makes the meta-score a robust indicator of reliability.

Built-in Quality Control

Because EPP is rooted in logistic regression, researchers can utilize "deviance"—a statistical measure of fit—to evaluate the ranking's validity. If the EPP scores show poor fit, it indicates that the models' performances are too erratic to be ranked reliably, providing a necessary safeguard against over-interpreting benchmark results.

Universal Interpretability

EPP scores are directly convertible back into winning probabilities. If Model A has a higher EPP than Model B, the framework allows for the calculation of exactly how often Model A is expected to win. Furthermore, the meta-score is "metric agnostic," allowing researchers to rank models based on accuracy, inference speed, or any other metric on the same comparable scale.

Empirical Validation

The EPP meta-score was validated using two major benchmarking environments:

  1. OpenML: A comparison of five common classification algorithms (Logistic Regression, k-NN, Random Forest variants, and Gradient Boosting) across diverse datasets.
  2. VTAB (Visual Task Adaptation Benchmark): A large-scale evaluation of vision model adaptation across various tasks.

Conclusion and Future Work

EPP offers a statistically rigorous alternative to traditional benchmarking. While it requires multiple "rounds" (datasets or folds) to calculate meaningful probabilities and involves slightly more computational overhead than simple averaging, its benefits in interpretability and standardization are significant. By adopting the Unified Benchmark Ontology, the machine learning community can move toward more transparent and reliable reporting in massive, multi-task leaderboards.

Summary updated

Abstract

In benchmarking different machine learning models, we are seeing a real boom—in many applications such as computer vision or neural language processing, more and more challenges are being created. We are therefore collecting more and more data on the performance of individual models or architectures, but the question remains as to how to decide which model gives the best results and whether one model is significantly better than another. The most commonly used measures, such as AUC, accuracy, or RMSE, return a numerical assessment of how well the predictions of the selected model satisfy specific properties: they correctly assign the probability of belonging to the chosen class, they are not wrong in assigning the predicted class, or the difference between the predictions and the true values is not large. From an application point of view, however, we lack information: (i) what is the probability that a given model gets a better performance model than another; (ii) whether the differences we observe between models are statistically significant; (iii) in most cases, the values of the selected model performance metrics are incomparable between different datasets, i.e., how to compare a model’s AUC improvement by $0.01$ if for one dataset the best achieved AUC is of the order of $0.9$ and for the other $0.7$. To address these shortcomings, in an earlier paper we introduce a new meta-measure of model performance—EPP. It is inspired by the Elo ranking used in chess and other sports games. By comparing the rankings of two players and transforming them accordingly, we obtain information on the probability that one player is better than the other. EPP adapts this property to the specific conditions of benchmarks in machine learning but allows for universal application in many benchmarking schemes. We emphasize this by introducing a unified terminology, the Unified Benchmark Ontology, and the description of the new measure is given in these terms. Hence, models are referred to as players and model performance to score.
BibTeX
@inproceedings{gosiewska2022interpretable,
  author = {Alicja Gosiewska and Katarzyna Woznica and Przemyslaw Biecek},
  title = {{Interpretable Meta-Score for Model Performance: Extended Abstract}},
  booktitle = {Meta-Knowledge Transfer @ ECML/PKDD},
  year = {2022},
  url = {https://openreview.net/forum?id=YMCP2tmD0Z},
}
Figure