How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation

Swaroop Mishra, Anjana Arunkumar

TL;DR

This paper argues that standard model leaderboards, often based solely on accuracy, provide an inequitable evaluation of AI models. They introduce a novel method to probe and customize leaderboards by weighting samples based on difficulty, leading to a more realistic assessment of model performance and a significant reduction in pre-deployment development and testing effort for industry users.

What Problem Does This Paper Address?

The authors highlight a critical disconnect: models that excel on leaderboards often falter in real-world applications. This discrepancy stems from how leaderboards typically evaluate models, primarily using accuracy on benchmark datasets. This approach can be misleading because it doesn't account for the varying "difficulty" of data samples. For instance, a model might achieve high accuracy by mastering easy examples while failing on harder ones, or it might exploit spurious correlations (unintended biases) in the data that don't generalize. This leads to an overestimation of AI capabilities and costly failures when models are deployed. The paper questions whether these leaderboards offer an "equitable evaluation" and proposes a solution to make them more robust and reflective of real-world performance.

Methods

The core of the paper's contribution lies in its novel approach to evaluating model performance by considering sample difficulty.

  • Difficulty Probing: They introduce a task-agnostic method to probe leaderboards by weighting samples based on their perceived difficulty. This is achieved by analyzing factors like out-of-distribution (OOD) characteristics and spurious bias.
  • New Evaluation Metrics: To address the limitations of accuracy, they propose three new metrics:
  • WSBias (Weighted Spurious Bias): This metric assigns higher weights to "harder" samples, aiming to penalize models that rely on spurious correlations.
  • WOOD (Weighted OOD Score): This metric leverages semantic textual similarity (STS) to identify OOD characteristics and weights samples accordingly.
  • WMProb (Weighted Maximum Probability): This metric considers a model's confidence in its predictions, penalizing models that are confidently wrong, which is crucial for safety-critical applications.
  • Customizable Visual Analytics Tool: They developed a prototype tool that allows users to customize leaderboards by selecting different metrics, weighting schemes, and data splits. This tool helps users analyze model strengths and weaknesses based on their specific application needs.

Datasets

The experiments were conducted on two movie review datasets:

  • SST-2 (Stanford Sentiment Treebank v2): A dataset of short movie reviews, used for both training and evaluation (IID evaluation).
  • IMDb: A dataset of full-length movie reviews, used for OOD evaluation.

The authors also mention using a recent work on OOD robustness (Hendrycks et al. 2020) as a basis for their experimental setup.

Key Insights and Ideas

  • Leaderboards are Adversarially Attackable: The paper demonstrates that standard leaderboards can be "attacked" by strategically selecting or weighting samples, revealing that top-ranked models aren't always the best for real-world use.
  • Accuracy is a Poor Proxy for Real-World Performance: The proposed metrics consistently show different rankings than accuracy, highlighting that models excelling in accuracy might not generalize well or handle difficult cases effectively.
  • Transformers Struggle with "Easy" Samples: Surprisingly, transformer models, despite their overall strong performance, were found to perform poorly on "easy" samples, suggesting they might be over-reliant on spurious biases or suffer from catastrophic forgetting due to large training data.
  • Simpler Models Leverage Spurious Bias: Conversely, simpler models (like word-averaging methods) were found to be more adept at utilizing spurious bias to solve easy questions, but this also leads to their performance being heavily penalized by the proposed metrics.
  • Confidence Matters: For safety-critical applications, a model's confidence in its incorrect predictions is a significant factor, and WMProb addresses this by incorporating prediction confidence.

Key Results

  • Reduced Development/Testing Effort: A user study with 32 experts across nine industries found that the prototype tool reduced pre-deployment development and testing effort by an average of 41%.
  • Model Performance Inflation Reduction: The proposed metrics (WSBias, WOOD, WMProb) showed a significant reduction in model performance inflation compared to accuracy, with WSBias scores decreasing by 25%-63% (Algorithm 1) and 11%-40% (Algorithm 2) relative to accuracy.
  • Significant Ranking Changes: The metrics led to substantial changes in model rankings for 8 out of 10 models tested, indicating that accuracy-based rankings are not robust. For example, WOOD showed a 15% decrease in score values compared to accuracy.
  • Identification of Model Strengths/Weaknesses: The analysis provided preliminary insights into the strengths and weaknesses of 10 different models across various scenarios.

Limitations and Future Work

  • Model Ensembles: The authors note that model ensembles often dominate leaderboards and suggest that their future work will expand the analysis to include them.
  • Generalization to Other Domains: While the proposed metrics are task-agnostic, they plan to expand their equitable evaluation metrics to other domains like vision and speech.
  • Competitions: The authors believe their equitable evaluation metrics could be useful in competitions and plan to explore this further.
  • Further User Study Expansion: They aim to expand their user study to include more participants and potentially more complex scenarios.

Summary updated

Abstract

Models that top leaderboards often perform unsatisfactorily when deployed in real world applications; this has necessitated rigorous and expensive pre-deployment model testing. A hitherto unexplored facet of model performance is: Are our leaderboards doing equitable evaluation? In this paper, we introduce a task-agnostic method to probe leaderboards by weighting samples based on their `difficulty' level. We find that leaderboards can be adversarially attacked and top performing models may not always be the best models. We subsequently propose alternate evaluation metrics. Our experiments on 10 models show changes in model ranking and an overall reduction in previously reported performance -- thus rectifying the overestimation of AI systems' capabilities. Inspired by behavioral testing principles, we further develop a prototype of a visual analytics tool that enables leaderboard revamping through customization, based on an end user's focus area. This helps users analyze models' strengths and weaknesses, and guides them in the selection of a model best suited for their application scenario. In a user study, members of various commercial product development teams, covering 5 focus areas, find that our prototype reduces pre-deployment development and testing effort by 41% on average.
BibTeX
@inproceedings{mishra2021how,
  author = {Swaroop Mishra and Anjana Arunkumar},
  title = {{How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=VB5bSci5mx},
}
Figure