Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports

Baoyu Jing, Zeya Wang, Eric P. Xing

Abstract

Chest X-Ray (CXR) images are essential for clinical screening and diagnosis. Automatically generating reports for these images can significantly reduce the workload of radiologists. However, the complex structural relationships within reports—specifically, the Impression section serving as a diagnostic summary of the Findings—and the overwhelming prevalence of normal observations pose significant challenges. We propose a novel framework that exploits these structures through a two-stage "Show, Describe and Conclude" strategy and a Cooperative Multi-Agent System (CMAS) designed to balance normal and abnormal descriptions. Our approach achieves state-of-the-art performance on major CXR datasets, demonstrating the value of integrating structural information into medical report generation.

Introduction

Generating medical reports from images is far more complex than standard image captioning. A typical chest X-ray report consists of two primary sections: Findings, which provides detailed anatomical descriptions, and Impression, which offers a final diagnostic summary. Existing AI models often struggle with two major hurdles: ignoring the structural hierarchy where the Impression depends on the Findings, and the "normality dominance" problem. In clinical practice, most X-rays are normal, leading to training data dominated by healthy descriptions. Consequently, standard models often fail to identify rare but critical abnormalities because they are "drowned out" by the sea of normality.

Methodology

Developed by researchers at Petuum Inc., our framework addresses these challenges through two core innovations.

Two-Stage Strategy: Show, Describe and Conclude

Instead of generating a report in a single pass, we implement a sequential workflow. The model first generates the Findings based on the input CXR image. Subsequently, it utilizes both the original image and the generated Findings to produce the Impression. This "Show, Describe and Conclude" approach ensures the final diagnosis is grounded in the descriptive evidence provided in the report.

The two-stage 'Show, Describe and Conclude' workflow.

Cooperative Multi-Agent System (CMAS)

To mitigate the imbalance between normal and abnormal observations, we decompose the writing process into a specialized multi-agent system:

  • The Planner (PL): Acts as the decision-maker, determining whether to continue writing and identifying if a specific image region is normal or abnormal.
  • The Normality Writer (NW): A specialized decoder trained specifically to generate common, standard descriptions of healthy anatomy.
  • The Abnormality Writer (AW): A specialized decoder focused solely on describing lesions and pathologies.
Architecture of the Cooperative Multi-Agent System (CMAS).

The system is optimized using Reinforcement Learning (REINFORCE) and Imitation Learning, allowing the agents to receive rewards based on the quality (e.g., BLEU-4 scores) of the generated text.

Results and Discussion

The proposed CMAS-RL model was evaluated on the IU-Xray and CX-CHR datasets, outperforming previous state-of-the-art methods across linguistic and clinical metrics.

  • Language Quality: On the CX-CHR dataset, the model achieved a BLEU-4 score of 0.545 and a CIDEr score of 2.900, significantly higher than baseline hierarchical models.
  • Clinical Accuracy: The model showed a marked improvement in abnormality detection. In the IU-Xray dataset, the CMAS-RL model achieved a precision of 0.128 for abnormality terms, nearly doubling the performance of several baseline models.
  • Specialization: By separating the NW and AW, the model prevents common "normal" templates from diluting the specific language required for rare diseases. Interestingly, the NW naturally learned to mimic standard templates used by radiologists for healthy patients without explicit programming.

Conclusion

By integrating structural information and utilizing specialized agents, our framework significantly improves the quality and clinical relevance of automated CXR reports. While the model occasionally misses secondary abnormalities or struggles with precise anatomical localization, it represents a significant step forward. Future work will explore incorporating explicit anatomical knowledge and 3D spatial reasoning to further refine report accuracy.

Summary updated

Abstract

Chest X-Ray (CXR) images are commonly used for clinical screening and diagnosis. Automatically writing reports for these images can considerably lighten the workload of radiologists for summarizing descriptive findings and conclusive impressions. The complex structures between and within sections of the reports pose a great challenge to the automatic report generation. Specifically, the section Impression is a diagnostic summarization over the section Findings; and the appearance of normality dominates each section over that of abnormality. Existing studies rarely explore and consider this fundamental structure information. In this work, we propose a novel framework that exploits the structure information between and within report sections for generating CXR imaging reports. First, we propose a two-stage strategy that explicitly models the relationship between Findings and Impression. Second, we design a novel cooperative multi-agent system that implicitly captures the imbalanced distribution between abnormality and normality. Experiments on two CXR report datasets show that our method achieves state-of-the-art performance in terms of various evaluation metrics. Our results expose that the proposed approach is able to generate high-quality medical reports through integrating the structure information.
BibTeX
@inproceedings{jing2019show,
  author = {Baoyu Jing and Zeya Wang and Eric P. Xing},
  title = {{Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=um6KG8IDa2Y},
}
Figure