Attention Based CNN-LSTM Network for Pulmonary Embolism Prediction on Chest Computed Tomography Pulmonary Angiograms

Sudhir Suman, Gagandeep Singh, Nicole Sakla, Rishabh Gattu, Jeremy Green, Tej Phatak, Dimitris Samaras, Prateek Prasanna

TL;DR

Researchers have developed a two-stage, attention-based CNN-LSTM network that mimics how radiologists analyze CT scans to detect Pulmonary Embolisms (PE). By capturing both the spatial details of individual slices and the sequential relationships between them, the model achieves a high AUC of 0.95, providing not just a "yes/no" diagnosis but also critical clinical details like the location and chronicity of the blood clots.

What Problem Does This Paper Address?

Pulmonary Embolism (PE)—a blood clot in the lung—is a leading cause of preventable hospital deaths. Diagnosing it via Computed Tomography Pulmonary Angiograms (CTPA) is a high-stakes, time-consuming task for radiologists, who must sift through hundreds of slices to find small, often subtle abnormalities.

Existing AI solutions often fall into two traps: they either treat every CT slice as an independent image (ignoring the 3D context) or they rely on computationally expensive "segmentation" methods that require pixel-perfect labeling. Furthermore, most models only provide a binary classification (PE present or absent), whereas doctors need to know the type (acute vs. chronic) and location (left, right, or central) to determine the best treatment.

Methods

The authors propose a two-stage architecture designed to mirror human cognitive processes:

  1. Stage 1: Spatial Feature Extraction: The model uses an EfficientNet CNN backbone to capture the visual properties of individual slices. To help the model "see" better, they pre-process the raw CT data into 3-channel RGB images, where each channel represents a different "window" (Lung, PE, and Mediastinal) optimized for different tissues.
  2. Stage 2: Sequential Reasoning: These features are fed into a Bidirectional LSTM (Bi-LSTM). This allows the network to understand the relationship between successive slices—essential because a clot is a 3D structure that spans multiple images.
  3. Attention Mechanism: The model uses a Multiple Instance Learning (MIL) pooling layer as an attention module. This automatically assigns higher weights to the "informative" slices (those likely to contain a clot) when making a final study-level prediction.
  4. Multi-Head Output: The network features two "heads": one for slice-level detection ($\psi_{image}$) and one for study-level characteristics ($\psi_{study}$), such as laterality and the Right Ventricular to Left Ventricular (RV/LV) ratio.
  5. Label Consistency: They implemented a custom loss function and logic rules to ensure the model doesn't make contradictory claims (e.g., it can't predict a "central PE" if it hasn't detected a PE in any individual slice).

Datasets

  • RSNA-STR PE (RSPECT) Dataset: The largest publicly available PE dataset, containing over 12,000 CT studies. The authors used 7,279 studies for training and validation.
  • Independent Test Set: An in-house curated dataset of 106 studies (56 positive, 50 negative) from a different clinical site to test the model's ability to generalize to new environments.

Key Insights and Ideas

  • Context is King: Treating CT slices as a sequence rather than independent images significantly boosts performance. The Bi-LSTM captures the "flow" of blood vessels across slices.
  • Attention as a Filter: PE often appears in only a tiny fraction of a 3D volume. The attention mechanism allows the model to ignore the "noise" of healthy tissue and focus its mathematical "gaze" on the suspicious regions.
  • Clinical Utility Beyond Binary: By predicting attributes like the RV/LV ratio, the model provides a proxy for heart strain, which is a critical factor in deciding how aggressively to treat a patient.
  • Visual Interpretability: Using Class Activation Maps (CAMs), the researchers showed that the model actually "looks" at the blood clots when making its decision, rather than relying on unrelated shortcuts in the data.

Key Results

  • High Accuracy: The proposed method achieved an AUC of 0.95 and an accuracy of 0.88 on the independent test set.
  • Superiority over Baselines: It significantly outperformed a standard CNN classifier (AUC 0.5) and a single-stage CNN-LSTM without attention (AUC 0.94).
  • Robustness: The model maintained high performance on an external dataset with different slice thicknesses and equipment, suggesting it is ready for real-world clinical diversity.

Limitations and Future Work

  • Bilateral PE Challenges: While the model is excellent at identifying clots on one side (unilateral), the Class Activation Maps were less precise at pinpointing the exact location of clots when they appeared on both sides (bilateral) of the lungs.
  • Future Directions: The authors plan to investigate end-to-end networks and the integration of "radiomic" features (hand-crafted mathematical descriptions of texture and shape) to further refine the prediction of PE characteristics.

Summary updated

Abstract

With more than 60,000 deaths annually in the United States, Pulmonary Embolism (PE) is among the most fatal cardiovascular diseases. It is caused by an artery blockage in the lung; confirming its presence is time-consuming and is prone to over-diagnosis. The utilization of automated PE detection systems is critical for diagnostic accuracy and efficiency. In this study we propose a two-stage attention-based CNN-LSTM network for predicting PE, its associated type (chronic, acute) and corresponding location (leftsided, rightsided or central) on computed tomography (CT) examinations. We trained our model on the largest available public Computed Tomography Pulmonary Angiogram PE dataset (RSNA-STR Pulmonary Embolism CT (RSPECT) Dataset, N = 7279 CT studies) and tested it on an in-house curated dataset of N = 106 studies. Our framework mirrors the radiologic diagnostic process via a multi-slice approach so that the accuracy and pathologic sequela of true pulmonary emboli may be meticulously assessed, enabling physicians to better appraise the morbidity of a PE when present. Our proposed method outperformed a baseline CNN classifier and a single-stage CNN-LSTM network, achieving an AUC of 0.95 on the test set for detecting the presence of PE in the study.
BibTeX
@inproceedings{suman2021attention,
  author = {Sudhir Suman and Gagandeep Singh and Nicole Sakla and Rishabh Gattu and Jeremy Green and Tej Phatak and Dimitris Samaras and Prateek Prasanna},
  title = {{Attention Based CNN-LSTM Network for Pulmonary Embolism Prediction on Chest Computed Tomography Pulmonary Angiograms}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=IuFZZ70gKb},
}
Figure