Introduction
Automated radiology report generation from medical images, such as chest X-rays, presents a unique challenge compared to standard image captioning. Radiologists do not merely describe visual elements; they interpret subtle cues using specialized domain knowledge. Current AI models often struggle with these nuances, medical jargon, and inherent data biases. Most medical datasets are unbalanced, frequently over-representing common pathologies while under-representing rare conditions or "normal" findings. This often leads to models that hallucinate abnormalities or overlook critical "disease-free" observations essential for a complete clinical picture.
To address these challenges, we present FODA-PG (Fine-grained Organ-Disease Adaptive Partitioning Graph), a framework designed to explicitly distinguish between normal and abnormal medical findings to produce more accurate and clinically coherent narratives.
The FODA-PG Framework
The core of the FODA-PG framework lies in its ability to partition radiological findings into granular "disease-specific" and "disease-free" categories. The architecture is divided into three primary modules:
- Graph Construction: The model builds a granular graph of radiological findings. It utilizes BioMedCLIP for retrieving relevant reference reports and NLTK for entity extraction. These entities are then partitioned into disease-specific and disease-free nodes. Node embeddings are generated via a modified DistilGPT2.
- Graph-Enhanced Visual Representation: A Vision Transformer (ViT) from MedSAM, fine-tuned on medical imagery, serves as the image encoder. A Graph-Enhanced Attention (GEA) mechanism aligns specific visual regions with the nodes in the organ-disease graph, ensuring the model focuses on clinically relevant areas.
- Graph-Guided Text Generation: Enhanced visual features are processed by a BiLSTM encoder, followed by a DistilGPT2 decoder for report generation. The model is further optimized using Reinforcement Learning (RL) to maximize clinical relevance, specifically targeting CIDEr scores.
Experimental Evaluation
The FODA-PG model was evaluated on two benchmark datasets: IU-Xray and MIMIC-CXR. The latter is a massive dataset of chest X-rays with associated clinical narratives, providing a robust test for clinical accuracy.
Key Results
FODA-PG consistently outperformed state-of-the-art (SOTA) models, including R2Gen, CMN, and DCL, across both Natural Language Generation (NLG) and Clinical Efficacy (CE) metrics.
- Natural Language Generation (NLG): The model achieved superior scores in BLEU, ROUGE-L, METEOR, and CIDEr, indicating high fluency and similarity to human-written reports.
- Clinical Efficacy (CE): On the MIMIC-CXR dataset, the model demonstrated significantly better performance in correctly identifying the presence or absence of specific pathologies, as shown in the CE metrics comparison.
- Ablation Studies: Removing the graph component or the adaptive partitioning led to a noticeable performance drop, confirming that the fine-grained partitioning is critical for success.
Discussion and Future Directions
The success of FODA-PG highlights the importance of treating "disease-free" attributes with equal weight to pathological ones. By structuring medical knowledge into a graph and using MedSAM for feature extraction, the model learns the complex relationships between organs and diseases, acting as a powerful prior during generation.
However, some limitations remain. The computational intensity of training with 8 NVIDIA A100 GPUs and complex attention mechanisms is high. Additionally, while effective for 2D chest X-rays, future work should explore scaling this adaptive partitioning approach to 3D modalities like CT or MRI scans. Further investigation into different embedding models, such as PubMedBERT, may also yield improvements as dataset sizes grow.
Summary updated