How Robust are Model Rankings : A Leaderboard Customization Approach for Equitable Evaluation

Swaroop Mishra, Anjana Arunkumar

Introduction

In the current landscape of Artificial Intelligence, leaderboards serve as the primary metric for progress. However, these rankings are often brittle, relying on aggregate accuracy scores that treat every test sample as equally significant. This "SOTA-chasing" culture overlooks a critical reality: a model might dominate a leaderboard by mastering "easy" samples—those with spurious biases or high similarity to training data—while failing on "hard," out-of-distribution (OOD) cases.

This paper introduces a framework to "probe" leaderboards by weighting samples based on difficulty, revealing that top-ranked models are often less robust than they appear. By introducing metrics like WSBias, WOOD, and WMProb, the authors provide a mechanism for equitable evaluation that aligns model selection with specific real-world requirements.

Quantifying Sample Difficulty

The authors propose three novel metrics to move beyond simple accuracy:

  1. WSBias (Weighted Spurious Bias): Identifies samples where models might "cheat" using unintended correlations (spurious biases).
  2. WOOD (Weighted OOD Score): Measures the semantic distance between test samples and the training set (e.g., SST-2 vs. IMDb), prioritizing models that generalize well to unfamiliar data.
  3. WMProb (Weighted Maximum Probability): A confidence-based metric that penalizes models more heavily for being "confidently wrong" than for being "humbly wrong."
Leaderboard customization across different industrial domains.

The core of this approach is a Leaderboard Customization Tool. This visual analytics prototype allows stakeholders to assign weights to data splits based on their specific focus area. As illustrated in Figure 1, a single accuracy-based ranking can be transformed into multiple customized rankings tailored for Healthcare, Manufacturing, or Conversational AI, ensuring the selected model meets the specific safety or performance needs of that domain.

Key Insights and the "Transformer Paradox"

The study's evaluation of ten popular NLP models yielded surprising results. When difficulty weighting was applied, 8 out of 10 models changed their ranking positions. Most notably, using the WOOD (OOD-focused) metric caused 9 out of 10 models to shift.

A significant finding is what the authors term the "Transformer Paradox." While large models like BERT and RoBERTa often lead in aggregate accuracy, they sometimes struggle with "easy" samples that simpler models handle with ease. This is attributed to "catastrophic forgetting," where the pursuit of complex generalization leads to the loss of simpler patterns. In several OOD scenarios, simpler architectures like Word2Vec-CNN actually outperformed massive Transformers.

Experimental Results

The proposed metrics provide a much-needed "performance correction." Standard accuracy was found to overestimate model capability by 11% to 63% compared to the equitable metrics. For example, the WSBias metric reduced reported scores by 25-63%, offering a more realistic view of how models handle biased data.

Comparison of standard accuracy against proposed equitable metrics across various NLP models.

As shown in the comparative analysis in Figure 3, while accuracy (blue line) remains high and relatively stable across models, the weighted metrics (yellow/red lines) fluctuate significantly. This gap reveals the underlying vulnerabilities of supposedly superior models and highlights how standard metrics can mask poor performance on difficult or biased samples.

Industry Impact and User Study

To validate the practical utility of the customization framework, the authors conducted a user study with 32 experts across five sectors. The results were compelling: the customization prototype reduced pre-deployment development and testing effort by an average of 41%.

The tool helped experts identify that models like RoBERTa-Large, despite having the highest number of correct predictions, distribute errors uniformly across confidence levels. This makes them potentially riskier for safety-critical applications compared to models with better-calibrated confidence that "know when they don't know."

Conclusion

The current reliance on static, accuracy-based leaderboards is insufficient for responsible AI deployment. By weighting samples by difficulty and allowing for domain-specific customization, we can achieve a more equitable and robust evaluation of AI models. Future work will look to expand these metrics into computer vision and speech, ensuring that "state-of-the-art" truly translates to "ready for the real world."

Summary updated

Abstract

Models that top leaderboards often perform unsatisfactorily when deployed in real world applications; this has necessitated rigorous and expensive pre-deployment model testing. A hitherto unexplored facet of model performance is: Are our leaderboards doing equitable evaluation? In this paper, we introduce a task-agnostic method to probe leaderboards by weighting samples based on their `difficulty' level. We find that leaderboards can be adversarially attacked and top performing models may not always be the best models. We subsequently propose alternate evaluation metrics. Our experiments on 10 models show changes in model ranking and an overall reduction in previously reported performance -- thus rectifying the overestimation of AI systems' capabilities. Inspired by behavioral testing principles, we further develop a prototype of a visual analytics tool that enables leaderboard revamping through customization, based on an end user's focus area. This helps users analyze models' strengths and weaknesses, and guides them in the selection of a model best suited for their application scenario. In a user study, members of various commercial product development teams, covering 5 focus areas, find that our prototype reduces pre-deployment development and testing effort by 41% on average.
BibTeX
@inproceedings{mishra2021how,
  author = {Swaroop Mishra and Anjana Arunkumar},
  title = {{How Robust are Model Rankings : A Leaderboard Customization Approach for Equitable Evaluation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=VB5bSci5mx},
}
Figure