Abstract
Baoyu Jing, Pengtao Xie, Eric Xing. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
Researchers have developed a multi-task, hierarchical deep learning framework that automatically generates detailed medical reports from images like chest X-rays. By combining a "co-attention" mechanism—which looks at both image regions and predicted medical tags—with a two-tiered LSTM, the model can produce long-form, clinically relevant paragraphs that outperform standard image captioning methods.
Introduction
Writing medical imaging reports is a high-stakes, labor-intensive task. For experienced radiologists, it represents a tedious bottleneck; for those less experienced or working in under-resourced areas, the process is prone to error. While "image captioning" (e.g., "a dog catching a frisbee") has seen significant progress, medical reports present three unique hurdles:
- Heterogeneity: A report consists of multiple components, including "findings" (detailed observations), "impressions" (conclusions), and "tags" (keywords).
- Localization: Abnormalities are often subtle and easily missed by global image encoders.
- Length: Medical reports are long, multi-sentence paragraphs. Standard Recurrent Neural Networks (RNNs) struggle to maintain coherence over such long sequences.
Methodology
The authors propose a multi-task hierarchical model with co-attention. The architecture is structured into three main stages:
- Tag Prediction (Multi-label Classification): The model utilizes a CNN (VGG-19) to extract visual features and predict a set of relevant medical tags (e.g., "effusion," "cardiomegaly"). These tags provide high-level semantic context that raw pixels alone might lack.
- Co-Attention Mechanism: This is the paper's core innovation. Instead of relying solely on the image, the model simultaneously "attends" to both visual image patches and the semantic embeddings of the predicted tags. This grounding helps the model align its descriptions with both visual evidence and clinical knowledge.
- Hierarchical LSTM: To handle long-form text, the decoder is split into two levels: a Sentence LSTM that determines the "topic" of the next sentence and controls when to terminate the report, and a Word LSTM that generates the specific sequence of words for that sentence.
Evaluation and Results
The model was evaluated on two publicly available datasets: IU X-Ray (7,470 chest X-ray pairs with multi-sentence reports) and PEIR Gross (7,442 pathology image-caption pairs).
The proposed model, Ours-CoAttention, significantly outperformed state-of-the-art image captioning baselines across all metrics:
- IU X-Ray (Paragraph Generation): The model achieved a BLEU-4 score of 0.247, nearly doubling the performance of standard Soft-Attention (0.118) and beating the "Ours-no-Attention" baseline (0.224).
- PEIR Gross (Single Sentence): The model demonstrated robustness across different imaging modalities (radiology vs. pathology), leading the pack in single-sentence generation.
- Qualitative Success: In side-by-side comparisons, the model correctly identified and described specific abnormalities like "calcified granuloma" or "degenerative changes of the spine," whereas baseline models often defaulted to generic "normal" reports.
Discussion
Key Insights
- Tags as Semantic Anchors: Visual attention alone often fails in medical contexts because interpretation is heavily influenced by clinical knowledge. Predicting tags first provides a "semantic prior" that guides visual attention to the correct regions.
- Hierarchy for Paragraph Coherence: Standard LSTMs tend to "forget" context in long sequences. The hierarchical structure allows the model to maintain a high-level plan (Sentence LSTM) while focusing on local syntax (Word LSTM).
- Synergy of Modalities: Visual features help localize where an abnormality is, while semantic tags help define what it is.
Limitations and Future Work
Despite its success, the model faces challenges such as sensitivity to image quality (e.g., misidentifying "eventration" as "interstitial opacities" in dark images) and error propagation from the initial tag prediction stage. Furthermore, while BLEU/ROUGE scores are promising, future work requires direct clinical validation by medical professionals to ensure diagnostic accuracy and safety.
Summary updated