Introduction
The rapid evolution of Large Language Models (LLMs) like GPT-4 and LLaMA has outpaced traditional Natural Language Processing (NLP) benchmarks. Metrics such as GLUE and SuperGLUE, once the gold standard, are now easily surpassed, leaving a fragmented landscape of thousands of disparate evaluation tasks. This paper addresses the inadequacy of current benchmarks in capturing "emergent abilities"—complex skills that appear only at scale—and the lack of a systematic link between these tasks and real-world model utility.
To bridge this gap, the authors propose a "Core Competency" framework, borrowing from corporate recruitment methodologies. Instead of focusing on isolated scores, this approach groups tasks into structured domains to assess a model's underlying capabilities.
The Core Competency Architecture
The proposed architecture organizes LLM evaluation into four primary pillars:
- Knowledge: Encompassing linguistic (grammar, semantics) and world knowledge (common sense, domain-specific).
- Reasoning: Covering causal, deductive, inductive, abductive, analogical, and multi-hop reasoning.
- Reliability: Focusing on truthfulness (hallucination) and calibration (uncertainty estimation).
- Safety: Addressing harmful content, social bias, and privacy.
Mapping the Benchmarking Landscape
The survey categorizes over 540 tasks to provide a roadmap for researchers. Key benchmarks are mapped to specific competencies to help identify which models excel in particular areas.
Knowledge and Reasoning Benchmarks
Linguistic knowledge is often evaluated through datasets like BLiMP, while world knowledge is tested via WikiFact or TruthfulQA. Reasoning, considered an "advanced" ability, is measured through complex tasks like GSM8K for mathematics or COPA for causal reasoning.
Key Insights and Ideas
- Emergent Abilities: Reasoning and tool manipulation exhibit non-linear performance curves, emerging sharply once models exceed specific parameter thresholds.
- Reliability vs. Knowledge: A model's vast knowledge does not guarantee reliability. "Linguistic calibration"—the ability to verbally express uncertainty—is a distinct and critical competency that often degrades after Reinforcement Learning from Human Feedback (RLHF).
- The "Fraud" Problem: Similar to job applicants memorizing answers, LLMs suffer from training set leakage. The authors argue for dynamic evaluation sets that are updated regularly to maintain validity.
Conclusion and Future Directions
While LLMs have largely mastered linguistic tasks, they continue to struggle with extrinsic hallucinations and complex planning. Future evaluation frameworks must focus on Planning (goal-oriented decision-making) and Coding (self-evolving capabilities). Furthermore, there is an urgent need for "clean" evaluation data to mitigate the effects of data leakage from massive web-crawled training sets.
Summary updated