Abstract
Baoyu Jing, Zeya Wang, Eric Xing. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
2019.
Introduction
Writing radiology reports is a high-stakes, time-consuming task that requires precision and a deep understanding of medical context. While artificial intelligence has made significant strides in image captioning, chest X-ray (CXR) reports present two unique challenges that standard models often fail to address. First, reports possess a complex sectional structure, typically divided into a detailed Findings section and a concise diagnostic Impression. Most existing models treat report generation as a single long-form captioning task, ignoring the hierarchical relationship where the Impression summarizes the Findings. Second, clinical datasets suffer from extreme data imbalance, where "normal" observations outnumber "abnormal" ones by factors as high as 12 to 1. This often leads models to default to generic descriptions of healthy anatomy, missing critical but rare pathologies.
The Cooperative Multi-Agent System (CMAS)
To address these challenges, this paper proposes a novel "Show, Describe, and Conclude" strategy powered by a Co-operative Multi-Agent System (CMAS). This framework mimics the structured workflow of human radiologists by decomposing the generation process into specialized tasks handled by three distinct agents:
- Planner (PL): Acts as the central manager, examining the image to decide which specialized writer to invoke next or when to conclude the report.
- Normality Writer (NW): Specialized in generating fluent, accurate descriptions of healthy anatomical features.
- Abnormality Writer (AW): Specialized in the nuanced medical language required to describe lesions, effusions, and other pathologies.
Workflow and Communication
The generation process occurs in two stages. First, the Findings are generated as the Planner iterates through image regions, tasking either the NW or AW to produce specific sentences. Second, the Impression is generated through a multi-modal summarization task, using both the original image and the newly generated Findings as input. A shared "Global State" is maintained to ensure the agents communicate effectively, preventing contradictions—such as one agent describing a normal heart size while another identifies cardiomegaly.
Training Strategy
The CMAS model is trained using a two-phase approach:
- Imitation Learning: The agents are pre-trained using standard cross-entropy loss to establish a baseline linguistic capability.
- Reinforcement Learning (RL): The system is fine-tuned using the REINFORCE algorithm. Agents receive rewards based on the BLEU-4 score of the generated report relative to the ground truth. This encourages the Planner to optimize its decision-making regarding which specialized writer to deploy for a given image region.
Experimental Results
The researchers evaluated the CMAS model on two major datasets: the public IU-Xray collection and CX-CHR, a large proprietary Chinese dataset. The model consistently outperformed state-of-the-art baselines across several dimensions:
- Automated Metrics: On the CX-CHR dataset, the model achieved a significant jump in CIDEr scores (measuring human-like description quality), rising from 1.58 in standard CNN-RNN models to 2.90 with CMAS-RL.
- Clinical Accuracy: The model demonstrated higher precision and lower False Positive Rates (FPR) for abnormality detection compared to previous methods.
- Qualitative Success: Human-like evaluations confirmed that the model successfully identified major abnormalities, such as cardiomegaly and calcified granulomas, which simpler models frequently overlooked.
Conclusion
By explicitly modeling the structural relationship between Findings and Impressions and employing specialized agents to handle the data imbalance between normal and abnormal observations, the CMAS framework represents a significant advancement in automated medical reporting. While the model occasionally struggles with secondary findings or subtle anatomical locations, its ability to produce clinically coherent and accurate reports marks a major step toward assisting radiologists in high-pressure clinical environments.
Summary updated