Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models

Mazda Moayeri, Vidhisha Balachandran, Varun Chandrasekaran, Safoora Yousefi, Thomas Fel, Soheil Feizi, Besmira Nushi, Neel Joshi, Vibhav Vineet

Introduction: The Plateau of Aggregate Accuracy

The evaluation of frontier foundation models has reached a "plateau of aggregate accuracy." Top-tier models like GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet frequently score within a fraction of a percentage point of each other on popular benchmarks. This convergence on leaderboards creates a misleading impression of model parity. In reality, these aggregate scores average away critical differences in strengths and weaknesses. Traditional benchmarks often test multiple capabilities—such as visual perception, mathematical calculation, and domain-specific knowledge—simultaneously. When a model fails, aggregate metrics offer no insight into why.

Methodology: Rationale-Parsed Skill Extraction

To move beyond blunt accuracy scores, we introduce an automated framework to extract fine-grained "skills" from model-generated rationales. Our pipeline, Rationale-Parsed Skill Extraction, consists of four key stages:

  1. Rationale Generation: A high-performing "annotator" model (GPT-4o) generates a step-by-step solution for each evaluation instance.
  2. Skill Tagging: The model labels the specific skill applied in each step of the rationale (e.g., "treble clef recognition" or "molar mass calculation").
  3. Skill-Slice Curation: Using text embeddings, we cluster these skills to handle naming variations and group instances from across 12 different benchmarks into unified "skill-slices."
  4. Verification: We validate the relevance of these extracted skills through post-hoc verification by independent models and human experts.

This methodology was applied to a corpus of 46,000 instances across 12 major benchmarks, including multimodal tasks like MMBench and SEEDBench, and language-centric ones like MMLU Pro.

Key Insights: Beyond the Leaderboard

Our analysis reveals that models with nearly identical aggregate scores possess vastly different capability profiles.

Deceptive Aggregate Metrics

While overall accuracies might differ by as little as 0.4%, performance on specific skill-slices can vary by as much as 20%. For instance, Gemini 1.5 Pro excels at chemistry-related calculations but lags significantly in legal reasoning compared to its peers. These "hidden" performance gaps suggest that model selection should be driven by specific task requirements rather than general leaderboard rankings.

Figure 1: Aggregate accuracy vs. fine-grained skill-slice performance.

Lopsided Evolution and Cross-Modality

We observed that model families evolve in distinct directions. Recent updates to the GPT and Claude families showed massive leaps in legal reasoning, while Gemini's improvements were concentrated in math and science. Furthermore, we found that skills are often modality-agnostic; a model's proficiency in "logical deduction" on a multimodal benchmark correlates strongly with its performance on the same skill in a text-only context. This suggests that these skills are deep latent properties of the model's reasoning engine.

Figure 5: Evolution of model performance on legal procedures across releases.

Key Results and Practical Applications

The practical utility of skill-level insights is most evident in "intelligent routing." By predicting which skills a new question requires, tasks can be routed to the model statistically most likely to succeed.

  • Performance Gaps: In the "computing molar mass" slice, Gemini 1.5 Pro was 18% more accurate than GPT-4o. In "applying constitutional law," GPT-4o was 19% more accurate than Gemini.
  • Routing Gains: Implementing a skill-based routing scheme improved overall accuracy by 3% across the 12-dataset corpus, with gains as high as 7% on MMLU Pro.
  • Retrieval Precision: Using skill annotations for retrieval (finding similar questions based on required skills) outperformed standard embedding-based methods by 18-50%.

Conclusion and Future Work

Moving beyond aggregate scores to skill-level analysis provides a more actionable understanding of foundation models. While the current framework relies on the quality of the annotator model and analyzes skills in isolation, it paves the way for more nuanced deployment strategies. Future work will explore "skill-interactions"—how the difficulty of a task changes when multiple complex skills must be used in tandem—and the scalability of automated annotation for increasingly difficult future benchmarks.

Summary updated

BibTeX
@inproceedings{moayeri2025unearthing,
  author = {Mazda Moayeri and Vidhisha Balachandran and Varun Chandrasekaran and Safoora Yousefi and Thomas Fel and Soheil Feizi and Besmira Nushi and Neel Joshi and Vibhav Vineet},
  title = {{Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=kNHVViEPWK},
}
Figure