Abstract
Evaluating long-form clinical AI responses is a high-stakes bottleneck because medical experts are expensive and often disagree. We introduce LONGQAEVAL, a framework demonstrating that fine-grained, sentence-level evaluation significantly improves inter-annotator agreement on factual correctness. Furthermore, we show that "partial" annotations—evaluating as few as three sentences—can provide reliable quality scores at approximately half the cost of full-text review.
Introduction
As large language models (LLMs) are increasingly integrated into healthcare, ensuring the correctness, relevance, and safety of their outputs is paramount. However, evaluating long-form clinical text presents significant challenges: expert scarcity, low inter-annotator reliability, and a "length bias" where longer responses are often unfairly favored in holistic evaluations. This paper investigates how different evaluation granularities and sampling strategies affect the reliability and efficiency of clinical QA assessment.
Methodology
We conducted a randomized human annotation study involving six licensed physicians. The study utilized 100 patient queries from the K-QA dataset, with 300 total answers generated by physicians, GPT-4, and Llama-3.1-Instruct-405B.
Two primary evaluation designs were compared:
- Coarse-grained: Annotators rated the entire answer on a 5-point Likert scale for Correctness, Relevance, and Safety.
- Fine-grained: Annotators rated individual sentences within the context of the full answer.
To address resource constraints, we explored partial annotations, testing whether evaluating a subset of sentences could approximate full-text scores. We also evaluated LLM-as-judge performance using GPT-4o with specialized annotation codebooks.
Results
Reliability and Granularity
Our findings indicate that the optimal granularity for evaluation depends on the dimension being measured. Fine-grained, sentence-level checks are significantly more reliable for Correctness, as they force raters to verify specific claims. As shown in Figure 1, the inter-annotator agreement (Randolph’s κ) for Correctness improved from 0.74 in coarse settings to 0.90 in fine-grained settings.
Conversely, Relevance is better assessed at a coarse level, as it is a holistic property of how a response addresses a patient's intent. Safety remains a difficult dimension to evaluate reliably, with low agreement across all granularities, suggesting that safety rubrics require more specialized development.
The Efficiency of Partial Sampling
A key contribution of this work is the "Three-Sentence" efficiency rule. We found that evaluating just three sentences per answer yields a reliability score (Spearman’s correlation > 0.8) comparable to evaluating the entire response.
This strategy effectively halves the time and cost of expert evaluation. While a full fine-grained evaluation took approximately 460 seconds per answer (compared to 239 seconds for coarse), the 3-sentence sampling strategy brings the effort back to levels comparable to coarse evaluation without sacrificing the reliability gains of fine-grained metrics.
Mitigating Verbosity Bias
Coarse evaluations often suffer from length bias, where models like GPT-4 receive higher scores simply for being more detailed. Fine-grained evaluation "levels the playing field," revealing that concise physician answers are often just as accurate as longer model-generated responses when judged sentence-by-sentence. This suggests that fine-grained metrics provide a more objective assessment of model capability versus human expertise.
Discussion and Conclusion
LONGQAEVAL provides a roadmap for more reliable and cost-effective clinical AI evaluation. By shifting to fine-grained correctness checks and utilizing partial sampling, researchers can achieve high-quality benchmarks even with limited expert resources. Furthermore, our results show that GPT-4o, when provided with fine-grained instructions, can achieve agreement with experts comparable to expert-expert agreement, suggesting a path forward for automated clinical evaluation. Future work should explore these methodologies in specialized medical domains and refine safety metrics for high-stakes clinical interactions.
Summary updated