Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions

Sebastian Heineking, Jonas Probst, Daniel Steinbach, Martin Potthast, Harrisen ScellsarXiv 2024 · arXiv

TL;DR

Evaluating how well Large Language Models (LLMs) answer open-ended health questions is notoriously difficult and expensive. This paper introduces Normalized Rank Position (NRP), a method that uses high-performing retrieval models to rank LLM-generated answers against a backdrop of expert-annotated documents. The authors demonstrate that this automated approach correlates strongly with human medical experts (Kendall’s $\tau = 0.64$), providing a scalable way to benchmark model performance and prompting strategies.

What Problem Does This Paper Address?

In high-stakes domains like healthcare, the "correctness" of an answer isn't a binary choice. Open-ended questions—the kind users actually ask search engines—require nuanced, credible, and readable responses.

Standard NLP metrics like BLEU or ROUGE (which look for word overlap) fail to capture these qualitative dimensions. While human expert evaluation remains the "gold standard," it is prohibitively slow and expensive to scale. The researchers sought a middle ground: an automated metric that captures expert-level nuance without requiring a doctor to grade every single model output.

Methods

The authors propose Normalized Rank Position (NRP). The core idea is to treat an LLM's answer as just another document in a search result.

  1. The Ranker: They first identify a state-of-the-art retrieval model (in this case, monoT5) that is already good at ranking human-written documents according to expert judgements (relevance, readability, and credibility).
  2. The Ranking: For a given health query, they take the LLM-generated answer and a set of human-written documents that already have expert labels. The retrieval model then ranks all of them together.
  3. The Score: The LLM's score is determined by where its answer lands in that list. If the model ranks the LLM answer at the very top (position 0), it gets an NRP of 1.0. If it's at the bottom, it gets a 0.0.

This allows researchers to see if an LLM answer is "better" (according to the ranker) than the documents experts originally deemed high-quality.

Datasets

The study utilized the CLEF eHealth 2021 dataset, which contains 55 health-related queries. Because of API changes at Reddit and Twitter, the authors focused on the 6,692 web documents within the collection, extracting plain text to ensure the retrieval models could process them fairly alongside the LLM outputs.

Key Insights and Ideas

  • Rankers as Proxies: High-quality retrieval models (like monoT5) trained on human text retain their effectiveness when judging LLM-generated text, despite differences in structure and length.
  • The Prompting Power-Up: Prompting strategy is often more important than raw model size. As seen in Figure 1, smaller models using the specialized MultiMedQA prompt often outperformed larger models that were given no context.
  • Saturation Points: While quality generally scales with parameters, the researchers observed a "saturation" in the Llama-2 family for this specific task; the 7B and 13B versions performed similarly, suggesting that for consumer health, bigger isn't always significantly better once a certain threshold is met.
  • The "Ceiling" Effect: Top-tier models like ChatGPT are so effective that they consistently rank above almost all human-written documents in the dataset, leading to an NRP near 1.0. This makes it difficult for NRP to distinguish between two "excellent" models.
Fig. 1. NRP of LLM answers, averaged over the ten gen- erated answers per question, grouped by prompt. Points are single answers. GPT-2 L and Llama-2 7B not shown.

Key Results

  • Expert Agreement: In a user study with a medical doctor, the NRP method achieved a mean correlation of $\tau = 0.64$. The expert and the automated ranker generally agreed that ChatGPT was the best, followed by Llama-2 13B, with GPT-2 variants at the bottom.
  • Scaling Trends: Figure 2 highlights a clear upward trend in answer quality as model parameters increase, particularly when moving from the GPT-2 era to modern instruction-tuned models like Falcon and Llama.
  • Top Performer: ChatGPT (GPT-3.5) was the clear winner, achieving a near-perfect mean NRP of 0.998.
Fig. 2. Number of model parameters vs. NRP using the MultiMedQA prompt.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Data Dependency: NRP requires a pre-existing collection of queries and expert-annotated documents. This makes it hard to apply to low-resource languages or brand-new domains where such datasets don't exist.
  • Model Bias: The evaluation is only as good as the retrieval model used. If the ranker is biased toward specific keywords or lengths, the NRP will reflect that bias.
  • Single Expert: The validation study relied on one medical professional; future work should involve a broader panel of experts to ensure the correlation holds across different subjective viewpoints.
  • RAG Systems: The authors suggest extending NRP to evaluate Retrieval-Augmented Generation (RAG) systems, checking if the generated answers accurately reflect the documents they cite.

Summary updated

Abstract

Evaluating the output of generative large language models (LLMs) is challenging and difficult to scale. Many evaluations of LLMs focus on tasks such as single-choice question-answering or text classification. These tasks are not suitable for assessing open-ended question-answering capabilities, which are critical in domains where expertise is required. One such domain is health, where misleading or incorrect answers can have a negative impact on a user's well-being. Using human experts to evaluate the quality of LLM answers is generally considered the gold standard, but expert annotation is costly and slow. We present a method for evaluating LLM answers that uses ranking models trained on annotated document collections as a substitute for explicit relevance judgements and apply it to the CLEF 2021 eHealth dataset. In a user study, our method correlates with the preferences of a human expert (Kendall's $τ=0.64$). It is also consistent with previous findings in that the quality of generated answers improves with the size of the model and more sophisticated prompting strategies.
BibTeX
@misc{heineking2024ranking,
  author = {Sebastian Heineking and Jonas Probst and Daniel Steinbach and Martin Potthast and Harrisen Scells},
  title = {{Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions}},
  year = {2024},
  eprint = {2408.09831},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.09831},
}
Figure