FODA-PG for Enhanced Medical Imaging Narrative Generation: Adaptive Differentiation of Normal and Abnormal Attributes

Kai Shu, Yuzhuo Jia, Ziyang Zhang, Jiechao Gao

Introduction

Automated radiology report generation from medical images, such as chest X-rays, presents a unique challenge compared to standard image captioning. Radiologists do not merely describe visual elements; they interpret subtle cues using specialized domain knowledge. Current AI models often struggle with these nuances, medical jargon, and inherent data biases. Most medical datasets are unbalanced, frequently over-representing common pathologies while under-representing rare conditions or "normal" findings. This often leads to models that hallucinate abnormalities or overlook critical "disease-free" observations essential for a complete clinical picture.

To address these challenges, we present FODA-PG (Fine-grained Organ-Disease Adaptive Partitioning Graph), a framework designed to explicitly distinguish between normal and abnormal medical findings to produce more accurate and clinically coherent narratives.

The FODA-PG Framework

The core of the FODA-PG framework lies in its ability to partition radiological findings into granular "disease-specific" and "disease-free" categories. The architecture is divided into three primary modules:

The FODA-PG architecture consists of three modules: (a) Graph Construction, (b) Graph-Enhanced Visual Representation, and (c) Graph-Guided Text Generation.
  1. Graph Construction: The model builds a granular graph of radiological findings. It utilizes BioMedCLIP for retrieving relevant reference reports and NLTK for entity extraction. These entities are then partitioned into disease-specific and disease-free nodes. Node embeddings are generated via a modified DistilGPT2.
  2. Graph-Enhanced Visual Representation: A Vision Transformer (ViT) from MedSAM, fine-tuned on medical imagery, serves as the image encoder. A Graph-Enhanced Attention (GEA) mechanism aligns specific visual regions with the nodes in the organ-disease graph, ensuring the model focuses on clinically relevant areas.
  3. Graph-Guided Text Generation: Enhanced visual features are processed by a BiLSTM encoder, followed by a DistilGPT2 decoder for report generation. The model is further optimized using Reinforcement Learning (RL) to maximize clinical relevance, specifically targeting CIDEr scores.

Experimental Evaluation

The FODA-PG model was evaluated on two benchmark datasets: IU-Xray and MIMIC-CXR. The latter is a massive dataset of chest X-rays with associated clinical narratives, providing a robust test for clinical accuracy.

Key Results

FODA-PG consistently outperformed state-of-the-art (SOTA) models, including R2Gen, CMN, and DCL, across both Natural Language Generation (NLG) and Clinical Efficacy (CE) metrics.

Comparison of Clinical Efficacy (CE) metrics on the MIMIC-CXR dataset, showing FODA-PG (labeled as Ours DCG) outperforming baseline and SOTA methods in Precision, Recall, and F1.
  • Natural Language Generation (NLG): The model achieved superior scores in BLEU, ROUGE-L, METEOR, and CIDEr, indicating high fluency and similarity to human-written reports.
  • Clinical Efficacy (CE): On the MIMIC-CXR dataset, the model demonstrated significantly better performance in correctly identifying the presence or absence of specific pathologies, as shown in the CE metrics comparison.
  • Ablation Studies: Removing the graph component or the adaptive partitioning led to a noticeable performance drop, confirming that the fine-grained partitioning is critical for success.

Discussion and Future Directions

The success of FODA-PG highlights the importance of treating "disease-free" attributes with equal weight to pathological ones. By structuring medical knowledge into a graph and using MedSAM for feature extraction, the model learns the complex relationships between organs and diseases, acting as a powerful prior during generation.

However, some limitations remain. The computational intensity of training with 8 NVIDIA A100 GPUs and complex attention mechanisms is high. Additionally, while effective for 2D chest X-rays, future work should explore scaling this adaptive partitioning approach to 3D modalities like CT or MRI scans. Further investigation into different embedding models, such as PubMedBERT, may also yield improvements as dataset sizes grow.

Summary updated

Abstract

Automatic Medical Imaging Narrative generation aims to alleviate the workload of radiologists by producing accurate clinical descriptions directly from radiological images. However, the subtle visual nuances and domain-specific terminology in medical images pose significant challenges compared to generic image captioning tasks. Existing approaches often neglect the vital distinction between normal and abnormal findings, leading to suboptimal performance. In this work, we propose FODA-PG, a novel Fine-grained Organ-Disease Adaptive Partitioning Graph framework that addresses these limitations through domain-adaptive learning. FODA-PG constructs a granular graphical representation of radiological findings by separating disease-related attributes into distinct "disease-specific" and "disease-free" categories based on their clinical significance and location. This adaptive partitioning enables our model to capture the nuanced differences between normal and pathological states, mitigating the impact of data biases. By integrating this fine-grained semantic knowledge into a powerful transformer-based architecture and providing rigorous mathematical justifications for its effectiveness, FODA-PG generates precise and clinically coherent reports with enhanced generalization capabilities. Extensive experiments on the IU-Xray and MIMIC-CXR benchmarks demonstrate the superiority of our approach over state-of-the-art methods, highlighting the importance of domain adaptation in medical report generation.
BibTeX
@inproceedings{shu2024foda,
  author = {Kai Shu and Yuzhuo Jia and Ziyang Zhang and Jiechao Gao},
  title = {{FODA-PG for Enhanced Medical Imaging Narrative Generation: Adaptive Differentiation of Normal and Abnormal Attributes}},
  booktitle = {BIBM},
  year = {2024},
  url = {https://openreview.net/forum?id=9J8E5LIQJh},
}
Figure