TRRG: Towards Truthful Radiology Report Generation With Cross-modal Disease Clue Enhanced Large Language Model

Yuhao Wang, Chao Hao, Yawen Cui, Xinqi Su, Weicheng Xie, Tao Tan, Zitong YuarXiv 2024 · DOI · arXiv

Introduction

Generating automated radiology reports is a high-stakes application of vision-language modeling. While standard models excel at general image captioning, medical imaging presents a significant challenge: data imbalance. Most radiology reports consist of "normal" descriptions, while critical abnormal findings—the actual diseases—are sparse. Consequently, standard models often suffer from coarse-grained alignment, failing to link specific visual lesions to medical terminology. This leads to reports that are fluent but lack "clinical truthfulness," often overlooking subtle but vital disease indicators in favor of generic descriptions.

To address this, the TRRG (Towards Truthful Radiology Report Generation) framework enhances Large Language Models (LLMs) with explicit "disease clues." This approach ensures the model prioritizes clinical accuracy alongside linguistic fluency.

Methodology

TRRG utilizes a two-stage training process designed to sharpen the model's perception of pathological features before and during the report generation phase.

The two-stage training framework of TRRG, involving disease-aware pre-training and clue-enhanced fine-tuning.

Stage 1: Disease-aware Pre-training

In the first stage, the framework employs contrastive learning (similar to the CLIP architecture) to align radiographs with specific sentences from medical reports. This stage "sharpens" the vision encoder’s ability to perceive fine-grained disease details, ensuring that the visual features extracted are clinically relevant before they are passed to the LLM.

Stage 2: Clue-Enhanced Fine-tuning

The core innovation of TRRG lies in its second stage, which introduces a novel mechanism for injecting medical knowledge directly into the generation process.

Detailed architecture of the TRRG fine-tuning stage, highlighting the Clue Injection and Cross-modal Clue Interaction modules.

This stage consists of three key components:

  1. Clue Injection Module: Using a library of disease templates (e.g., [Severity] [Disease] at [Location]), the system generates "clue embeddings" via a frozen text encoder.
  2. Cross-modal Clue Interaction: The model employs a two-stream attention mechanism. One stream processes visual features while the other processes the textual disease clues. Learnable queries help the model dynamically select the most relevant clues for a specific image.
  3. Disease-aware Consistency Loss (DAL): This specialized loss function mathematically forces visual features and textual disease clues to align, ensuring the model's internal representation "sees" the pathology it is describing.

Key Insights and Results

The researchers found that a "Top-K" Filter was essential; selecting only the top three most likely disease clues based on visual weights yielded the best results, as too many clues could confuse the model. This explicit injection of disease-oriented information effectively overcomes the "normal" bias inherent in medical datasets.

TRRG was evaluated on the IU-Xray and MIMIC-CXR benchmarks, achieving State-of-the-Art (SOTA) performance on the latter:

  • Language Fluency: Achieved a BLEU-4 score of 0.157 and a ROUGE-L of 0.336.
  • Clinical Effectiveness (CE): Using the ChexBERT automated labeler, TRRG reached an F1 score of 0.393, significantly outperforming previous transformer-based architectures.
  • Qualitative Performance: Case studies demonstrated that TRRG could correctly identify conditions like "probable atelectasis" and "small bilateral pleural effusions" in instances where baseline models incorrectly reported the lungs as "clear."

Conclusion and Future Work

TRRG represents a significant step toward clinically truthful AI in radiology. By providing the LLM with a "cheat sheet" of medical knowledge through clue injection, the model moves beyond simple pattern matching. Future research will focus on generalizing this approach to more complex modalities such as CT, MRI, and Ultrasound, and addressing the model's current sensitivity to smaller datasets.

Summary updated

Abstract

The vision-language modeling capability of multi-modal large language models has attracted wide attention from the community. However, in medical domain, radiology report generation using vision-language models still faces significant challenges due to the imbalanced data distribution caused by numerous negated descriptions in radiology reports and issues such as rough alignment between radiology reports and radiography. In this paper, we propose a truthful radiology report generation framework, namely TRRG, based on stage-wise training for cross-modal disease clue injection into large language models. In pre-training stage, During the pre-training phase, contrastive learning is employed to enhance the ability of visual encoder to perceive fine-grained disease details. In fine-tuning stage, the clue injection module we proposed significantly enhances the disease-oriented perception capability of the large language model by effectively incorporating the robust zero-shot disease perception. Finally, through the cross-modal clue interaction module, our model effectively achieves the multi-granular interaction of visual embeddings and an arbitrary number of disease clue embeddings. This significantly enhances the report generation capability and clinical effectiveness of multi-modal large language models in the field of radiology reportgeneration. Experimental results demonstrate that our proposed pre-training and fine-tuning framework achieves state-of-the-art performance in radiology report generation on datasets such as IU-Xray and MIMIC-CXR. Further analysis indicates that our proposed method can effectively enhance the model to perceive diseases and improve its clinical effectiveness.
BibTeX
@misc{wang2024trrg,
  author = {Yuhao Wang and Chao Hao and Yawen Cui and Xinqi Su and Weicheng Xie and Tao Tan and Zitong Yu},
  title = {{TRRG: Towards Truthful Radiology Report Generation With Cross-modal Disease Clue Enhanced Large Language Model}},
  year = {2024},
  eprint = {2408.12141},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.12141},
}
Figure