TL;DR
Evaluating how well Large Language Models (LLMs) answer open-ended health questions is notoriously difficult and expensive. This paper introduces Normalized Rank Position (NRP), a method that uses high-performing retrieval models to rank LLM-generated answers against a backdrop of expert-annotated documents. The authors demonstrate that this automated approach correlates strongly with human medical experts (Kendall’s $\tau = 0.64$), providing a scalable way to benchmark model performance and prompting strategies.
What Problem Does This Paper Address?
In high-stakes domains like healthcare, the "correctness" of an answer isn't a binary choice. Open-ended questions—the kind users actually ask search engines—require nuanced, credible, and readable responses.
Standard NLP metrics like BLEU or ROUGE (which look for word overlap) fail to capture these qualitative dimensions. While human expert evaluation remains the "gold standard," it is prohibitively slow and expensive to scale. The researchers sought a middle ground: an automated metric that captures expert-level nuance without requiring a doctor to grade every single model output.
Methods
The authors propose Normalized Rank Position (NRP). The core idea is to treat an LLM's answer as just another document in a search result.
- The Ranker: They first identify a state-of-the-art retrieval model (in this case, monoT5) that is already good at ranking human-written documents according to expert judgements (relevance, readability, and credibility).
- The Ranking: For a given health query, they take the LLM-generated answer and a set of human-written documents that already have expert labels. The retrieval model then ranks all of them together.
- The Score: The LLM's score is determined by where its answer lands in that list. If the model ranks the LLM answer at the very top (position 0), it gets an NRP of 1.0. If it's at the bottom, it gets a 0.0.
This allows researchers to see if an LLM answer is "better" (according to the ranker) than the documents experts originally deemed high-quality.
Datasets
The study utilized the CLEF eHealth 2021 dataset, which contains 55 health-related queries. Because of API changes at Reddit and Twitter, the authors focused on the 6,692 web documents within the collection, extracting plain text to ensure the retrieval models could process them fairly alongside the LLM outputs.
Key Insights and Ideas
- Rankers as Proxies: High-quality retrieval models (like monoT5) trained on human text retain their effectiveness when judging LLM-generated text, despite differences in structure and length.
- The Prompting Power-Up: Prompting strategy is often more important than raw model size. As seen in Figure 1, smaller models using the specialized
MultiMedQAprompt often outperformed larger models that were given no context. - Saturation Points: While quality generally scales with parameters, the researchers observed a "saturation" in the Llama-2 family for this specific task; the 7B and 13B versions performed similarly, suggesting that for consumer health, bigger isn't always significantly better once a certain threshold is met.
- The "Ceiling" Effect: Top-tier models like ChatGPT are so effective that they consistently rank above almost all human-written documents in the dataset, leading to an NRP near 1.0. This makes it difficult for NRP to distinguish between two "excellent" models.
Key Results
- Expert Agreement: In a user study with a medical doctor, the NRP method achieved a mean correlation of $\tau = 0.64$. The expert and the automated ranker generally agreed that ChatGPT was the best, followed by Llama-2 13B, with GPT-2 variants at the bottom.
- Scaling Trends: Figure 2 highlights a clear upward trend in answer quality as model parameters increase, particularly when moving from the GPT-2 era to modern instruction-tuned models like Falcon and Llama.
- Top Performer: ChatGPT (GPT-3.5) was the clear winner, achieving a near-perfect mean NRP of 0.998.
Limitations and Future Work
The authors acknowledge several hurdles:
- Data Dependency: NRP requires a pre-existing collection of queries and expert-annotated documents. This makes it hard to apply to low-resource languages or brand-new domains where such datasets don't exist.
- Model Bias: The evaluation is only as good as the retrieval model used. If the ranker is biased toward specific keywords or lengths, the NRP will reflect that bias.
- Single Expert: The validation study relied on one medical professional; future work should involve a broader panel of experts to ensure the correlation holds across different subjective viewpoints.
- RAG Systems: The authors suggest extending NRP to evaluate Retrieval-Augmented Generation (RAG) systems, checking if the generated answers accurately reflect the documents they cite.
Summary updated