HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants

Milan Gritta, Gerasimos Lampouras, Ignacio Iacobacci

TL;DR

Evaluating language models (LMs) as conversational assistants currently requires either expensive human judges or biased, costly "LLM-as-a-judge" setups. This paper introduces HumanRankEval (HRE), a fast, cheap, and fully automatic evaluation method that ranks human-authored answers based on an LM's internal log-likelihoods and correlates them with actual human votes. HRE achieves a remarkable 0.96 correlation with human preferences on Chatbot Arena while remaining highly sensitive to the effects of instruction tuning.

What Problem Does This Paper Address?

How do we know if a language model will make a good conversational assistant? Currently, the gold standard is human evaluation (like Chatbot Arena), but it is slow, expensive, and difficult to scale.

To bypass this, the community often relies on two alternatives:

  1. Knowledge-based benchmarks (e.g., multiple-choice QA), which fail to capture open-ended conversational nuances and instruction-following behavior.
  2. LLM-as-a-judge (e.g., using GPT-4 to grade outputs), which introduces systemic biases, data privacy concerns, high API costs, and reproducibility issues due to shifting proprietary models.

The authors address this bottleneck by proposing an automatic evaluation metric that directly measures how well an open-source LM's internal "preferences" align with real human preferences, without needing an external LLM judge or human annotators at inference time.

Methods

The core idea behind HumanRankEval is elegant: if an LM is well-aligned with humans, it should assign higher probabilities to answers that humans prefer.

Figure 1: Overview of HumanRankEval: given a ques- tion with multiple answers, we correlate human scores of each answer with the log-likelihoods of the LM. The unabridged answers can be found in Figure 2.

The process works as follows:

  1. Prompting: The LM is fed a question ($Q$) along with a set of four different human-authored answers ($A_1$ to $A_4$).
  2. Log-Likelihood Calculation: Instead of generating text, the method extracts the model's logits to compute the log-likelihood of each answer sequence, normalized by character length to avoid favoring longer responses.
  3. Correlation: The model's internal ranking of the answers (based on log-likelihood) is compared against the actual human rankings (based on community votes) using Pearson correlation.
  4. Scoring: The correlation coefficients are micro-averaged across thousands of questions to produce the final HRE score.

The authors explicitly choose Pearson correlation over Spearman rank correlation because human vote counts are continuous rather than strictly ordinal, which empirically leads to a much clearer separation of model capabilities.

Datasets

The authors introduced a new, high-quality QA dataset curated from StackExchange and StackOverflow.

Figure 2: HumanRankEval example from StackOverflow (Java topic).

The dataset consists of 7,000 questions (yielding 28,000 total answers) evenly split across 14 diverse, objective topics ranging from programming languages (Python, Java, C++) to technical and academic fields (Physics, LaTeX, Statistics).

To ensure the human rankings were reliable, the data was rigorously filtered:

  • Each question must have exactly 4 answers.
  • The answers must have been collectively rated by at least 40 people (averaging 10+ votes per answer) to guarantee a robust consensus.
  • Every individual answer must have at least 5 votes to filter out low-quality spam.

Key Insights and Ideas

  • Log-Likelihood as a Preference Proxy: An LM's token-level probability distribution implicitly contains a ranking of text quality. We don't need to make a model generate text to evaluate its conversational alignment; we can simply look at what it "expects" a good answer to look like.
  • The Benchmark Blindspot: Popular leaderboards like OpenLLM often fail to capture the massive behavioral shift that occurs during instruction tuning. Because they focus on raw knowledge retrieval, they can score a base model higher than its chat-tuned variant. HRE fixes this by focusing purely on the style and preference of the response.
  • Pearson > Spearman for Rankings: Treating human votes as continuous values (Pearson) rather than simple ranks (Spearman) preserves the magnitude of human preference, providing a more stable and monotonic metric for model scaling.

Key Results

  • High Human Correlation: HRE achieves a 0.96 Pearson correlation with the human-voted Chatbot Arena leaderboard. This outperforms MT-Bench (0.92) and drastically beats OpenLLM (0.85).
  • Clear Model Scaling: HRE scores scale predictably and monotonically with model size. The authors demonstrated this across the Pythia family (70M to 12B parameters) and the StarCoderBase family (1B to 15.5B parameters).
Figure 5: HumanRankEval (avg) scores for Pythia LMs.
  • Sensitivity to Instruction Tuning: Unlike knowledge benchmarks, HRE clearly differentiates between base models and their instruction-tuned counterparts. For example, it correctly identifies that Llama2-Chat-7B possesses superior instruction-following capabilities compared to the larger, un-tuned Llama2-13B base model—an alignment shift that OpenLLM completely misses.

Limitations and Future Work

  • Composite Attribute: HRE evaluates human preference as a single, holistic score. It cannot break down why a model prefers an answer (e.g., separating factual correctness from helpfulness, safety, or tone).
  • Language Bias: The dataset is currently restricted entirely to English-language content.
  • White-Box Requirement: Because HRE relies on extracting log-likelihoods from model logits, it cannot be easily used to evaluate closed-source, black-box APIs (like GPT-4 or Claude) that do not expose token-level probabilities.
  • Contamination Risks: Since StackOverflow and StackExchange data are frequently scraped for LLM pretraining, there is a risk of data contamination, though the authors note that instruction-tuning on overlapping QA pairs poses a much higher risk of score inflation than raw web pretraining.

Summary updated

Abstract

Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation of such LMs would ideally be performed using human judgement, however, this is not scalable. On the other hand, automatic evaluation featuring auxiliary LMs as judges and/or knowledge-based tasks is scalable but struggles with assessing conversational ability and adherence to instructions. To help accelerate the development of LMs as conversational assistants, we propose a novel automatic evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. To perform evaluation, HRE ranks these answers based on their log-likelihood under the LM's distribution, and subsequently calculates their correlation with the corresponding human rankings. We support HRE's efficacy by investigating how efficiently it separates pretrained and instruction-tuned LMs of various sizes. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.
BibTeX
@inproceedings{gritta2024humanrankeval,
  author = {Milan Gritta and Gerasimos Lampouras and Ignacio Iacobacci},
  title = {{HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.456},
}
Figure