ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment

Ruochen Li, Jun Li, Bailiang Jian, Kun Yuan, Youxiang ZhuEMNLP 2025

TL;DR

Current automated metrics for evaluating radiology reports often give high scores to AI-generated text that clinicians find untrustworthy. This paper introduces "ReEvalMed," a framework that defines clinically grounded criteria for evaluating these metrics, revealing significant flaws in existing approaches and offering a path toward more reliable medical report assessment.

What Problem Does This Paper Address?

Radiology reports are crucial for clinical decision-making, guiding diagnosis and treatment. With the rise of AI models that can generate these reports directly from medical images, evaluating their quality has become paramount. However, standard NLP metrics like BLEU and ROUGE-L, which focus on surface-level text similarity, often fail to align with real-world clinical judgment. AI-generated reports might score highly on these metrics but still contain factual inaccuracies, logical inconsistencies, or omissions that could compromise patient safety. This disconnect highlights a critical gap: current metrics don't truly capture what makes a medical report clinically useful and trustworthy. The paper aims to bridge this gap by proposing a more clinically relevant evaluation framework.

Methods

The core of this paper is the development of a "Meta-Evaluation framework" designed to assess existing evaluation metrics for medical reports. This framework is built upon two key dimensions:

  1. Alignment with Clinical Needs: This involves evaluating how well metrics capture essential clinical information such as accurate description, location, distance, and size of findings. It also considers aspects like negation, modality awareness, and the expression of uncertainty.
  2. Core Metric Capabilities: This dimension assesses fundamental properties of a good metric, including:
  3. Discriminative Ability: The metric's capacity to distinguish between clinically significant errors (which could lead to harm) and clinically insignificant ones.
  4. Robustness: The metric's tolerance for minor, clinically irrelevant variations in phrasing or structure that don't affect the report's meaning.
  5. Monotonicity: The metric's ability to consistently assign lower scores as the severity of clinical errors increases.

To test these capabilities, the researchers created a dataset of "Ground Truth-Meta-Evaluation (GT-ME) pairs." This dataset consists of original radiology reports (GT) and rewritten versions (ME) that introduce specific types of errors (omission, fabrication, inaccuracy) at varying levels of clinical significance. These pairs were meticulously annotated by clinicians with explanations for the error types and their significance.

Datasets

The study utilizes two primary sources of clinical data:

  • MIMIC-CXR: A publicly available dataset of de-identified chest X-ray reports.
  • ReXVal: A dataset specifically designed for evaluating radiology report generation, containing expert-annotated errors.

The researchers sampled 50 reports from MIMIC-CXR and 20 from ReXVal to construct their novel GT-ME report pairs dataset. This dataset comprises 400 expert-validated pairs, each designed to isolate and test specific clinical aspects and error types.

Key Insights and Ideas

  • The "Trust Gap" is Real and Quantifiable: The paper provides concrete evidence that standard NLP metrics fail to align with clinical judgment, leading to a disconnect between automated scores and actual report quality.
  • Clinical Semantics Matter More Than Surface Similarity: Metrics that only focus on word overlap or general text similarity miss crucial clinical nuances, such as the impact of misreporting lesion size or location.
  • Existing Metrics Struggle with Nuance: Many current metrics are too coarse-grained, failing to differentiate between minor stylistic variations and critical factual errors.
  • Clinician Expertise is Indispensable: The development of clinically grounded evaluation criteria, as proposed in this paper, requires deep collaboration with and insights from practicing clinicians.
  • A Multi-faceted Evaluation is Necessary: A single score is insufficient. Metrics need to be evaluated on multiple dimensions like discrimination, robustness, and monotonicity to truly reflect clinical utility.

Key Results

The paper's experiments revealed significant limitations in various evaluation metrics:

  • General NLP Metrics (BLEU, ROUGE-L, METEOR): These metrics showed poor robustness, frequently penalizing harmless stylistic variations. They also struggled with clinical relevance.
  • BERT-F1: While robust to minor variations, it failed to adequately penalize clinically significant errors, indicating a lack of sensitivity to critical issues.
  • AlignScore: Demonstrated a low discriminative score, suggesting it struggles to differentiate between clinically important and unimportant errors.
  • Domain-Specific Metrics (e.g., CheXbert-F1, RadGraph-F1): While some showed better discrimination, they often lacked robustness or were overly sensitive to specific phrasing, hindering their overall clinical applicability.
  • Monotonicity: Most metrics showed a decreasing trend with increasing error severity, but key transitions revealed weaknesses. For instance, many metrics failed to distinguish between minor factual errors and single clinically significant errors, and struggled with logical contradictions.

Limitations and Future Work

Limitations:

  • Dataset Scalability: The manual verification and expert annotation process for creating the GT-ME dataset is labor-intensive, limiting its scalability.
  • Metric Interpretability: While the framework defines criteria, it doesn't directly assess the interpretability of the metrics themselves, which is crucial for clinical users.
  • Modality Transferability: The current framework is primarily tested on CXR data; its generalizability to other imaging modalities (like CT or MRI) needs further validation.

Future Work:

  • Incorporating Clinical Knowledge: Future metrics could benefit from integrating domain-specific ontologies or structured clinical guidelines to better understand subtle clinical variations.
  • LLM-based Reasoning: Techniques like chain-of-thought prompting or agent-based debate could enhance LLM-based metrics' ability to assess semantic consistency and clinical implications.
  • Scalable Annotation Workflows: Developing more automated pipelines for generating and validating report variations, potentially using LLMs with selective expert review, could improve dataset scalability.
  • Broader Modality Support: Extending the evaluation framework and dataset to other medical imaging modalities.

Summary updated

BibTeX
@inproceedings{li2025reevalmed,
  author = {Ruochen Li and Jun Li and Bailiang Jian and Kun Yuan and Youxiang Zhu},
  title = {{ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure