TL;DR
This paper introduces Report Cards, an automated evaluation framework that generates structured, natural language summaries of large language model (LLM) behaviors across specific skills. Using an iterative generation method called PRESS, the authors demonstrate that qualitative summaries can accurately differentiate models, reflect real capability rankings, and offer human-interpretable insights beyond scalar benchmark scores.
What Problem Does This Paper Address?
Standard LLM evaluation relies on quantitative summary statistics (such as leaderboard accuracy) that obscure fine-grained failure modes, or raw output inspection that does not scale. There is currently no standardized, automated mechanism to produce holistic, qualitative diagnostics of model strengths and weaknesses. The authors tackle this gap by automating qualitative evaluations into concise, human-readable summaries.
Methods
The authors structure qualitative evaluation around three criteria and an iterative generation pipeline:
- Evaluation Dimensions:
- Specificity: Assessed via contrastive accuracy, where a guesser model must correctly match an anonymous set of completions to the corresponding model's Report Card.
- Faithfulness: Measured via the correlation ($R^2$) between Card Elo (pairwise preferences based strictly on reading two Report Cards) and Oracle Elo (pairwise preferences based on ground-truth answer correctness).
- Interpretability: Evaluated by human raters along dimensions of relevance, informativeness, and clarity.
- PRESS (Progressive Refinement for Effective Skill Summarization): An iterative algorithm that generates Report Cards without human supervision. It samples small batches ("quizzes") of question-answer pairs, produces intermediate behavioral summaries (progression), and iteratively merges them into a cohesive bulleted format while enforcing length constraints (refinement).
Datasets
The framework is evaluated across diverse domains:
- MMLU: Selected high school and college STEM and humanities subjects (e.g., Mathematics, Physics, Chemistry, Biology, Machine Learning).
- Anthropic Advanced AI Risk: Subsets evaluating model safety and alignment traits (Corrigibility and Myopic Reward).
- CN Grammar: A proprietary Chinese grammar error correction dataset used to evaluate open-ended linguistic editing.
Key Insights and Ideas
- Substance Over Style: When responses are de-stylized (paraphrased or stripped of reasoning artifacts), Report Cards maintain higher contrastive matching accuracy than few-shot output samples, confirming that they describe underlying capabilities rather than surface-level prose styles.
- Iterative Refinement Matters: Iterative updating via PRESS captures fine-grained failure patterns that one-pass full-context prompting tends to smooth over or omit.
- Efficient Skill Profiling: Pairwise Card Elo rankings closely mirror true model performance using significantly fewer direct model comparisons than standard per-instance Elo evaluations.
Key Results
- Specificity: PRESS achieves contrastive accuracies between 0.62 and 0.80 across MMLU and risk tasks, outperforming raw few-shot baselines on stylized MMLU benchmarks.
- Faithfulness: Card Elo achieves an $R^2$ of ~0.79 on MMLU against ground-truth Elo, surpassing both general Chatbot Arena ratings and sample-based few-shot Elo.
- Interpretability: Human annotators rated generated summaries above 4.0 on a 5-point Likert scale for clarity, relevance, and informativeness.
Limitations and Future Work
- Evaluator Reliance: Report Card quality relies on the capabilities and potential latent biases of the generator and judge models.
- Domain Breadth: Experiments focus primarily on multiple-choice and short-form datasets; open-ended generative domains (e.g., creative writing, open dialogue) remain unverified.
- Automated Scoring Gap: LLM-based proxy raters showed only weak-to-moderate correlation with human evaluators, requiring further work to reliably automate qualitative validation.
Summary updated