Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLM

Hongzhao Li, Hongyu Wang, Xia Sun, Hua He, Jun Feng

TL;DR

This paper introduces a novel approach to automatically generate structured and interpretable chest X-ray reports using a pre-trained Large Language Model (LLM). By combining anatomical region identification, prompt engineering, and clinical context, their method aims to improve report clarity, physician interactivity, and overall clinical relevance, addressing key limitations of existing automated radiology report generation systems.

What Problem Does This Paper Address?

Generating accurate and comprehensive radiology reports from medical images like chest X-rays is a critical but time-consuming task for physicians. Current automated report generation methods often produce unstructured text, making it difficult for clinicians to quickly extract essential information. Furthermore, these systems typically lack interpretability, meaning it's hard to understand why a particular finding was reported, and offer limited interactivity, preventing physicians from easily guiding the generation process based on patient-specific context. This opacity and rigidity hinder the practical application of these tools in clinical settings.

Methods

The proposed framework leverages a pre-trained LLM to generate structured chest X-ray reports. The core methodology involves several key steps:

  1. Anatomy Region Detection and Feature Extraction: They use a Faster R-CNN model to identify 29 specific anatomical regions within a chest X-ray and extract visual features for each.
  2. Sentence Generator: A Transformer Decoder model takes the extracted visual features and generates region-specific descriptive sentences, forming the initial foundation of the report.
  3. Anatomy Prompts Generation: This module converts the detected anatomical information into textual prompts. It includes classifiers for sentence detection (whether a region warrants a sentence) and abnormality detection (whether a region is abnormal). These are then translated into clear prompts like "The aortic arch is definitely abnormal."
  4. Structured Report Generation: The heart of the system is an LLM (specifically GPT-4) that receives a combination of inputs:
  5. Customized Prompt (C): A general instruction to generate a structured report.
  6. Generated Sentences (Y): The region-specific descriptions from the Sentence Generator.
  7. Anatomical Prompts (P1, P2): The prompts generated from anatomical region and abnormality detection.
  8. Clinical Context Prompts (P3): Patient-specific information such as medical history and reason for examination, provided by a physician.

By integrating these diverse inputs, the LLM synthesizes a comprehensive, clinically relevant, and structured report.

Datasets

The primary dataset used for training and evaluation is MIMIC-CXR [6]. This is described as the largest publicly available dataset of chest radiographs and associated reports, containing over 377,000 images and 227,000 reports. The authors followed standard data splits for training, validation, and testing. They also utilized labels generated by Chest ImaGenome [20] to align sentences with 29 specific chest anatomical regions.

Key Insights and Ideas

  • Anatomy-Centric Foundation: Building the report foundation on region-specific sentences is a novel contribution that directly addresses the "structural deficiency" problem.
  • Prompt Engineering for Structure and Interactivity: The paper highlights the power of prompt engineering, specifically using anatomical and clinical context prompts, to guide LLMs towards generating structured, interpretable, and interactive reports. This moves beyond simple end-to-end generation.
  • Physician-in-the-Loop: The inclusion of clinical context prompts (P3) explicitly allows physicians to participate in the report generation process, enabling adjustments based on real-world patient needs.
  • Interpretability through Color-Coding: The visualization in Figure 4 demonstrates how color-coding anatomical regions and their corresponding sentences enhances interpretability, allowing physicians to easily associate findings with specific image areas.
  • Trade-off between NLG and Clinical Accuracy: The authors note a slight decrease in some NLG metrics (like ROUGE-L) when incorporating more detailed anatomical information and clinical context, but this is justified by a significant improvement in clinical correctness and report structure.

Key Results

  • State-of-the-Art Performance: The proposed model achieved top performance across most Natural Language Generation (NLG) metrics (BLEU, METEOR) and significantly outperformed existing methods on Clinical Effectiveness (CE) metrics, demonstrating superior medical accuracy and reliability.
  • High Localization Accuracy: Table II shows high IoU scores for anatomical region detection across various regions, indicating precise identification.
  • Effective Sentence and Anomaly Detection: Table III shows promising results for sentence detection (F1: 0.701) and abnormality detection (F1: 0.557), with high recall for abnormality detection (0.951), crucial for clinical applications.
  • Qualitative Improvements: Figure 4 and the qualitative analysis demonstrate that the prompt-guided LLM generates more accurate, detailed, and structured reports compared to models without prompt integration, effectively incorporating clinical context and anatomical details.

Limitations and Future Work

  • Subjectivity of Normal Sentences: The authors acknowledge that the inclusion of "normal" sentences is subjective and can lead to a slight underrepresentation in metrics like ROUGE-L, as some physicians might omit normal findings.
  • NLG Metric Trade-off: While clinical accuracy is improved, there's a minor decrease in some NLG metrics, which the authors attribute to the increased detail and structure of their reports, making them more comprehensive but potentially less concise than some reference reports.
  • Future Work: The paper implicitly suggests future work could involve further refining the balance between NLG metrics and clinical accuracy, exploring more advanced LLM architectures, and potentially expanding the system to other imaging modalities or report types. The current integration module uses a fixed GPT-4, implying future work could explore fine-tuning LLMs for this specific task.

Summary updated

Abstract

Medical report generation automates radiology descriptions from images, easing the burden on physicians and minimizing errors. However, current methods lack structured outputs and physician interactivity for clear, clinically relevant reports. Our method introduces a prompt-guided approach to generate structured chest X-ray reports using a pre-trained large language model (LLM). First, we identify anatomical regions in chest X-rays to generate focused sentences that center on key visual elements, thereby establishing a structured report foundation with anatomy-based sentences. We also convert the detected anatomy into textual prompts conveying anatomical comprehension to the LLM. Additionally, the clinical context prompts guide the LLM to emphasize interactivity and clinical requirements. By integrating anatomy-focused sentences and anatomy/clinical prompts, the pre-trained LLM can generate structured chest X-ray reports tailored to prompted anatomical regions and clinical contexts. We evaluate using language generation and clinical effectiveness metrics, demonstrating strong performance.
BibTeX
@inproceedings{li2024prompt,
  author = {Hongzhao Li and Hongyu Wang and Xia Sun and Hua He and Jun Feng},
  title = {{Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLM}},
  booktitle = {ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=FkSQukMXt6},
}
Figure