Abstract
Chest X-Ray (CXR) images are essential for clinical screening and diagnosis. Automatically generating reports for these images can significantly reduce the workload of radiologists. However, the complex structural relationships within reports—specifically, the Impression section serving as a diagnostic summary of the Findings—and the overwhelming prevalence of normal observations pose significant challenges. We propose a novel framework that exploits these structures through a two-stage "Show, Describe and Conclude" strategy and a Cooperative Multi-Agent System (CMAS) designed to balance normal and abnormal descriptions. Our approach achieves state-of-the-art performance on major CXR datasets, demonstrating the value of integrating structural information into medical report generation.
Introduction
Generating medical reports from images is far more complex than standard image captioning. A typical chest X-ray report consists of two primary sections: Findings, which provides detailed anatomical descriptions, and Impression, which offers a final diagnostic summary. Existing AI models often struggle with two major hurdles: ignoring the structural hierarchy where the Impression depends on the Findings, and the "normality dominance" problem. In clinical practice, most X-rays are normal, leading to training data dominated by healthy descriptions. Consequently, standard models often fail to identify rare but critical abnormalities because they are "drowned out" by the sea of normality.
Methodology
Developed by researchers at Petuum Inc., our framework addresses these challenges through two core innovations.
Two-Stage Strategy: Show, Describe and Conclude
Instead of generating a report in a single pass, we implement a sequential workflow. The model first generates the Findings based on the input CXR image. Subsequently, it utilizes both the original image and the generated Findings to produce the Impression. This "Show, Describe and Conclude" approach ensures the final diagnosis is grounded in the descriptive evidence provided in the report.
Cooperative Multi-Agent System (CMAS)
To mitigate the imbalance between normal and abnormal observations, we decompose the writing process into a specialized multi-agent system:
- The Planner (PL): Acts as the decision-maker, determining whether to continue writing and identifying if a specific image region is normal or abnormal.
- The Normality Writer (NW): A specialized decoder trained specifically to generate common, standard descriptions of healthy anatomy.
- The Abnormality Writer (AW): A specialized decoder focused solely on describing lesions and pathologies.
The system is optimized using Reinforcement Learning (REINFORCE) and Imitation Learning, allowing the agents to receive rewards based on the quality (e.g., BLEU-4 scores) of the generated text.
Results and Discussion
The proposed CMAS-RL model was evaluated on the IU-Xray and CX-CHR datasets, outperforming previous state-of-the-art methods across linguistic and clinical metrics.
- Language Quality: On the CX-CHR dataset, the model achieved a BLEU-4 score of 0.545 and a CIDEr score of 2.900, significantly higher than baseline hierarchical models.
- Clinical Accuracy: The model showed a marked improvement in abnormality detection. In the IU-Xray dataset, the CMAS-RL model achieved a precision of 0.128 for abnormality terms, nearly doubling the performance of several baseline models.
- Specialization: By separating the NW and AW, the model prevents common "normal" templates from diluting the specific language required for rare diseases. Interestingly, the NW naturally learned to mimic standard templates used by radiologists for healthy patients without explicit programming.
Conclusion
By integrating structural information and utilizing specialized agents, our framework significantly improves the quality and clinical relevance of automated CXR reports. While the model occasionally misses secondary abnormalities or struggles with precise anatomical localization, it represents a significant step forward. Future work will explore incorporating explicit anatomical knowledge and 3D spatial reasoning to further refine report accuracy.
Summary updated