Image-aware Evaluation of Generated Medical Reports

Gefen Dawidowicz, Elad Hirsch, Ayellet Tal

Introduction

The evaluation of AI-generated medical reports is a critical bottleneck in the development of automated radiology assistants. Traditional Natural Language Generation (NLG) metrics like BLEU or ROUGE, while standard in general NLP, often fail in clinical settings because they prioritize word overlap over clinical accuracy. A report that swaps "left" for "right" might achieve a high BLEU score but represents a dangerous medical error. Conversely, Clinical Efficacy (CE) metrics like CheXpert are limited to a fixed set of pathologies and often miss nuances in location or severity.

To bridge this gap, researchers from Technion have introduced VLScore, a vision-language metric that incorporates the original X-ray image into the evaluation process. By measuring the geometric relationship between the image and the text in a shared embedding space, VLScore provides a more clinically grounded assessment of report quality.

The VLScore Methodology

VLScore moves beyond text-to-text comparison by utilizing a shared visual-textual embedding space, specifically leveraging the LIMITR model. The metric evaluates the relationship between three distinct entities:

  1. $i$: The embedding of the input X-ray image.
  2. $r$: The embedding of the ground-truth (reference) report.
  3. $g$: The embedding of the AI-generated (candidate) report.

The core innovation lies in treating these three embeddings as vertices of a triangle in a high-dimensional space. The quality of the generated report is determined by the area of the triangle formed by $(i, r, g)$.

The Logic of the Triangle

If a generated report is accurate, it should be positioned very close to both the reference report and the visual evidence in the image. A smaller triangle area indicates higher clinical alignment. This area is then normalized into a score between 0 and 1, where 1 represents a perfect match. Unlike simple cosine similarity, the triangle approach explicitly models the tripartite relationship, ensuring the generated text is grounded in both the gold-standard text and the actual image.

Experimental Setup

The researchers validated VLScore using two primary datasets:

  • ReXVal (Radiology Report Expert Evaluation): A benchmark of 200 report pairs evaluated by six board-certified radiologists. This serves as the gold standard for human-AI alignment.
  • Perturbed Dataset: A new dataset of 979 triplets introduced by the authors. In this set, reports were manually modified with either significant clinical errors (e.g., swapping anatomical locations like "basal" to "apical") or insignificant "fluff" changes (e.g., removing boilerplate phrases like "a chest radiograph was obtained"). This allowed the team to test whether a metric could distinguish between a harmless paraphrase and a medical error.

Key Findings and Results

Alignment with Experts

On the ReXVal dataset, VLScore achieved a Kendall rank correlation ($\tau$) of 0.718 with radiologist judgments. This represents a significant improvement over previous benchmarks: for comparison, BLEU-4 scored 0.345, and the previous state-of-the-art clinical metric, RadGraph F1, scored 0.515.

Sensitivity to Clinical Nuance

VLScore proved highly sensitive to clinical errors that traditional metrics missed. In tests involving anatomical swaps, VLScore showed the highest sensitivity ($\Delta = 0.19$), while word-overlap metrics showed almost zero change. Conversely, VLScore was robust to "equivalent normals"; it correctly assigned high scores to different ways of phrasing a normal scan, whereas BLEU-4 often penalized these variations due to low word overlap.

The Importance of the Image

Traditional metrics treat the reference report as the absolute truth. However, radiologists often include non-image information (like patient history) in their reports. By including the image in the metric calculation, VLScore focuses on what the AI should be able to see, effectively treating the image as the ultimate ground truth.

Limitations and Future Work

While VLScore represents a major advancement, its performance is inherently tied to the quality of the underlying multi-modal embedding model (LIMITR). As vision-language models improve, the accuracy of VLScore is expected to scale accordingly. Additionally, the normalization constant ($C$) used to scale the triangle area may require tuning when applied to different medical domains, such as MRIs or CT scans.

Finally, the authors emphasize that while VLScore is a leap forward for automated evaluation, it is intended to assist and streamline the development process, not to replace the final clinical judgment of human radiologists in real-world deployments.

Summary updated

BibTeX
@inproceedings{dawidowicz2024image,
  author = {Gefen Dawidowicz and Elad Hirsch and Ayellet Tal},
  title = {{Image-aware Evaluation of Generated Medical Reports}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=ecPIg6o84Z},
}
Figure