Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

Blair Yang, Fuyang Cui, Keiran Paster, Jimmy Ba, Pashootan Vaezipoor, Silviu Pitis, Michael R. ZhangSubmission to ICLR 2025 · Rejected

TL;DR

This paper introduces Report Cards, an automated evaluation framework that generates structured, natural language summaries of large language model (LLM) behaviors across specific skills. Using an iterative generation method called PRESS, the authors demonstrate that qualitative summaries can accurately differentiate models, reflect real capability rankings, and offer human-interpretable insights beyond scalar benchmark scores.

What Problem Does This Paper Address?

Standard LLM evaluation relies on quantitative summary statistics (such as leaderboard accuracy) that obscure fine-grained failure modes, or raw output inspection that does not scale. There is currently no standardized, automated mechanism to produce holistic, qualitative diagnostics of model strengths and weaknesses. The authors tackle this gap by automating qualitative evaluations into concise, human-readable summaries.

Methods

The authors structure qualitative evaluation around three criteria and an iterative generation pipeline:

  • Evaluation Dimensions:
  • Specificity: Assessed via contrastive accuracy, where a guesser model must correctly match an anonymous set of completions to the corresponding model's Report Card.
  • Faithfulness: Measured via the correlation ($R^2$) between Card Elo (pairwise preferences based strictly on reading two Report Cards) and Oracle Elo (pairwise preferences based on ground-truth answer correctness).
  • Interpretability: Evaluated by human raters along dimensions of relevance, informativeness, and clarity.
  • PRESS (Progressive Refinement for Effective Skill Summarization): An iterative algorithm that generates Report Cards without human supervision. It samples small batches ("quizzes") of question-answer pairs, produces intermediate behavioral summaries (progression), and iteratively merges them into a cohesive bulleted format while enforcing length constraints (refinement).

Datasets

The framework is evaluated across diverse domains:

  • MMLU: Selected high school and college STEM and humanities subjects (e.g., Mathematics, Physics, Chemistry, Biology, Machine Learning).
  • Anthropic Advanced AI Risk: Subsets evaluating model safety and alignment traits (Corrigibility and Myopic Reward).
  • CN Grammar: A proprietary Chinese grammar error correction dataset used to evaluate open-ended linguistic editing.

Key Insights and Ideas

  • Substance Over Style: When responses are de-stylized (paraphrased or stripped of reasoning artifacts), Report Cards maintain higher contrastive matching accuracy than few-shot output samples, confirming that they describe underlying capabilities rather than surface-level prose styles.
  • Iterative Refinement Matters: Iterative updating via PRESS captures fine-grained failure patterns that one-pass full-context prompting tends to smooth over or omit.
  • Efficient Skill Profiling: Pairwise Card Elo rankings closely mirror true model performance using significantly fewer direct model comparisons than standard per-instance Elo evaluations.

Key Results

  • Specificity: PRESS achieves contrastive accuracies between 0.62 and 0.80 across MMLU and risk tasks, outperforming raw few-shot baselines on stylized MMLU benchmarks.
  • Faithfulness: Card Elo achieves an $R^2$ of ~0.79 on MMLU against ground-truth Elo, surpassing both general Chatbot Arena ratings and sample-based few-shot Elo.
  • Interpretability: Human annotators rated generated summaries above 4.0 on a 5-point Likert scale for clarity, relevance, and informativeness.

Limitations and Future Work

  • Evaluator Reliance: Report Card quality relies on the capabilities and potential latent biases of the generator and judge models.
  • Domain Breadth: Experiments focus primarily on multiple-choice and short-form datasets; open-ended generative domains (e.g., creative writing, open dialogue) remain unverified.
  • Automated Scoring Gap: LLM-based proxy raters showed only weak-to-moderate correlation with human evaluators, requiring further work to reliably automate qualitative validation.

Summary updated

Abstract

The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.
BibTeX
@misc{yang2024report,
  author = {Blair Yang and Fuyang Cui and Keiran Paster and Jimmy Ba and Pashootan Vaezipoor and Silviu Pitis and Michael R. Zhang},
  title = {{Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries}},
  year = {2024},
  eprint = {2409.00844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.00844},
}
Figure