Subject2Vec: Generative-Discriminative Approach from a Set of Image Patches to a Vector

Sumedha Singla, Mingming Gong, Siamak Ravanbakhsh, Frank C. Sciurba, Barnabás Póczos, Kayhan N. Batmanghelich

TL;DR

This paper introduces Subject2Vec, a novel deep learning approach that learns a subject-level representation from sets of image patches for disease severity prediction. By combining discriminative, attention, and generative modules, it achieves state-of-the-art performance on COPD prediction while offering interpretable insights into which lung regions are most relevant to the disease.

What Problem Does This Paper Address?

Diagnosing and assessing the severity of diffuse lung diseases like Chronic Obstructive Pulmonary Disease (COPD) from medical images (e.g., CT scans) is challenging. Traditional deep learning methods often require fixed-size inputs, which is problematic for medical images that vary significantly in size and resolution. Furthermore, these methods can be "black boxes," making it difficult to understand why a particular prediction is made. This paper aims to address these limitations by developing a model that can handle variable-sized inputs (represented as sets of image patches) and provide interpretable results, highlighting the specific anatomical regions contributing to the disease assessment.

Methods

The core of Subject2Vec is a three-module architecture trained end-to-end:

  1. Discriminative Network: This module takes local image features (extracted from patches) and aggregates them into a fixed-length subject-level representation. This representation is then used to predict disease severity. It employs an encoder to extract patch features and a permutation-invariant function (like mean or max pooling) to combine them.
  2. Attention Mechanism: This module learns to assign weights to each image patch, indicating its importance for the disease severity prediction. This provides interpretability by highlighting critical regions. The attention network is designed to be permutation-equivariant, ensuring that the order of patches doesn't affect the output.
  3. Generative Network: This module, implemented as a convolutional auto-decoder (CAE), reconstructs the input patches from their latent representations. Its purpose is to regularize the encoder, encouraging it to learn diverse and non-redundant latent features. This helps prevent the attention mechanism from assigning uniform weights, which would hinder interpretability.

The overall objective function balances the discriminative loss (for accurate prediction), the generative loss (for feature diversity), and an attention regularization term.

Datasets

  • Synthetic Data: 10,000 training and 8,000 testing sets of MNIST images. The set-label was the sum of prime numbers within the set, used to evaluate the interpretability of attention weights as detectors.
  • COPDGene Dataset: A large-scale dataset of 6,400 subjects with varying degrees of COPD severity. This dataset was used to evaluate the model's performance in predicting clinical measures of COPD.

Key Insights and Ideas

  • Patch-based representation for variable input sizes: Treating subjects as sets of image patches elegantly sidesteps the need for image resizing, which can distort intensity values and affect interpretation in medical imaging.
  • Attention for interpretability: The attention mechanism is crucial for understanding which parts of the lung are most indicative of disease severity, moving beyond black-box predictions.
  • Generative regularization for diverse features: The generative component is not just about reconstruction; it actively encourages the encoder to learn a richer, more diverse set of latent features. This diversity is key to making the attention mechanism meaningful and preventing it from collapsing into uniform weights.
  • Balancing prediction and interpretability: The paper explicitly shows a trade-off: stronger generative regularization (leading to better interpretability) can slightly decrease predictive accuracy. This highlights a critical consideration in medical AI development.

Key Results

  • State-of-the-art performance: The Subject2Vec model achieved state-of-the-art results on the COPDGene dataset for predicting clinical measures of disease severity (FEV1, FEV1/FVC, and GOLD stage). For example, it achieved an accuracy of 61.17% for GOLD exact classification, outperforming several baseline methods.
  • Effective interpretability: On the synthetic dataset, the attention weights successfully acted as detectors for relevant instances (prime numbers), demonstrating the model's ability to learn from weak, set-level supervision.
  • Visualization of attention: Visualizations on COPD subjects showed that the attention maps highlighted regions of the lung that were clinically identified as severely damaged, correlating with higher disease severity scores.

Limitations and Future Work

  • Spatial location ignored: The current model treats patches as a set, disregarding their specific spatial relationships within the 3D volume. The authors acknowledge this as a limitation.
  • Future Direction: The authors plan to extend the model to incorporate spatial relationships between patches, which could further enhance its understanding of lung pathology.

Summary updated

Abstract

We propose an attention-based method that aggregates local image features to a subject-level representation for predicting disease severity. In contrast to classical deep learning that requires a fixed dimensional input, our method operates on a set of image patches; hence it can accommodate variable length input image without image resizing. The model learns a clinically interpretable subject-level representation that is reflective of the disease severity. Our model consists of three mutually dependent modules which regulate each other: (1) a discriminative network that learns a fixed-length representation from local features and maps them to disease severity; (2) an attention mechanism that provides interpretability by focusing on the areas of the anatomy that contribute the most to the prediction task; and (3) a generative network that encourages the diversity of the local latent features. The generative term ensures that the attention weights are non-degenerate while maintaining the relevance of the local regions to the disease severity. We train our model end-to-end in the context of a large-scale lung CT study of Chronic Obstructive Pulmonary Disease (COPD). Our model gives state-of-the art performance in predicting clinical measures of severity for COPD. The distribution of the attention provides the regional relevance of lung tissue to the clinical measurements.
BibTeX
@inproceedings{singla2018subject,
  author = {Sumedha Singla and Mingming Gong and Siamak Ravanbakhsh and Frank C. Sciurba and Barnabás Póczos and Kayhan N. Batmanghelich},
  title = {{Subject2Vec: Generative-Discriminative Approach from a Set of Image Patches to a Vector}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  doi = {10.1007/978-3-030-00928-1\_57},
}
Figure