Regularizing and Interpreting Vision Transformer by Patch Selection on Echocardiography Data

Alfred Nilsson, Hossein AzizpourCHIL 2024 · OpenReview

TL;DR

This paper introduces G-ViT, a novel method that uses stochastic patch selection during training to regularize Vision Transformers (ViTs) for echocardiography video analysis. This approach not only improves generalization and interpretability but also achieves state-of-the-art performance in predicting Left Ventricular Ejection Fraction (LVEF) while using only a fraction of the input data.

What Problem Does This Paper Address?

Predicting Left Ventricular Ejection Fraction (LVEF) from echocardiography videos is crucial for assessing cardiac function. However, this task faces challenges: echocardiography data is high-dimensional but often limited in scale, making models prone to overfitting. Traditional methods for analyzing these videos often involve complex pipelines with separate segmentation and prediction steps. While Vision Transformers (ViTs) have shown promise in computer vision, their application to small, high-dimensional datasets like echocardiography requires methods to enhance generalization and interpretability. The authors aim to address this by developing an end-to-end learned model that can effectively regularize ViTs and provide insights into their decision-making process.

Methods

The core of the proposed method, G-ViT, is the integration of stochastic embedded feature selection into a Video Vision Transformer (V-ViT) architecture.

  • V-ViT Architecture: This forms the baseline, comprising a ViT frame encoder for spatial feature extraction from individual video frames and a transformer sequence model for temporal aggregation. This factorization allows for patch-level selection.
  • Gumbel-Softmax (GS) Distribution: This is a differentiable sampling technique that allows for the selection of features (in this case, image patches) in a continuous, learnable manner. By using GS distributions, the model can learn to probabilistically select which patches are most informative.
  • Stochastic Patch Selection: Instead of processing all patches in a video frame, G-ViT learns to select a subset of patches for each frame during training. This is achieved by sampling from GS distributions, where the parameters of these distributions are learned. The temperature parameter of the GS distribution is annealed during training, encouraging the model to transition towards discrete patch selections.
  • Regularization: This stochastic patch selection acts as a form of regularization. By forcing the model to focus on a subset of patches, it prevents overfitting to noisy or less informative regions, thereby improving generalization.
  • Interpretability: The learned patch selections can be visualized, revealing which regions of the echocardiography video the model deems important for LVEF prediction. This provides insights into the model's decision-making process.

Datasets

  • EchoNet-Dynamic: This is the primary dataset used for training and evaluation. It consists of 10,030 labeled echocardiogram videos, each capturing several cardiac cycles with annotations for LVEF. The dataset is publicly available for academic and research purposes.

Key Insights and Ideas

  • Patch Selection as Regularization: The paper demonstrates that learning to select a subset of image patches during training is a powerful regularization technique for ViTs, especially in data-scarce, high-dimensional domains.
  • Interpretability through Selection: Visualizing the selected patches provides a direct window into what the model is "looking at," revealing clinically relevant regions like the left ventricle and areas of high motion. This offers a more intuitive understanding than traditional black-box model explanations.
  • Efficiency Gains: By selecting only a fraction of the input patches, the model can achieve comparable or better performance with significantly reduced computational load during training.
  • Robustness to Masking: The learned feature selection is more robust to increasing mask ratios (i.e., using fewer patches) compared to random masking, maintaining a non-trivial performance even when a large portion of the input is discarded.

Key Results

  • State-of-the-Art Performance: G-ViT achieved state-of-the-art performance on the EchoNet-Dynamic dataset for end-to-end LVEF prediction among learned methods, outperforming both a random selection baseline and the standard V-ViT.
  • Regularization Effect: With an optimal mask ratio (MR) of 0.3, G-ViT showed significant improvements across all metrics (RMSE, MAE, R²) compared to the baseline V-ViT trained on the full input.
  • Reduced Model Variance: G-ViT also demonstrated reduced model variance, indicated by smaller standard deviations in performance across different random seeds.
  • Efficiency: Remarkably, the G-ViT model using only 50% of the image patches achieved performance comparable to the full-input V-ViT.
  • Interpretability: Visualizations showed that G-ViT consistently prioritized regions of known clinical significance, such as the left ventricle and areas of high motion, aligning with human expert knowledge.

Limitations and Future Work

  • Hyperparameter Tuning: While the authors performed extensive hyperparameter searches, the optimal settings for mask ratio and temperature might be dataset-specific.
  • Generalizability: The current work focuses on echocardiography. Future work could explore the applicability of G-ViT to other medical imaging modalities or other domains with similar data characteristics (small-scale, high-dimensional).
  • Downstream Applications: The authors suggest that the method's ability to provide interpretable feature selections could extend beyond regularization to various downstream applications requiring explainable AI.
  • Further Exploration of Mask Ratios: While an optimal MR of 0.3 was found, further exploration of the trade-offs at different mask ratios could yield additional insights.

Summary updated

Abstract

This work introduces a novel approach to model regularization and explanation in \Glspl{vit}, particularly beneficial for small-scale but high-dimensional data regimes, such as in healthcare. We introduce stochastic embedded feature selection in the context of echocardiography video analysis, specifically focusing on the EchoNet-Dynamic dataset for the prediction of \gls{lvef}. Our proposed method, termed \Glspl{gvit}, augments \Glspl{vvit}, a performant transformer architecture for videos with \Glspl{cae}, a common dataset-level feature selection technique, to enhance \gls{vvit}’s generalization and interpretability. The key contribution lies in the incorporation of stochastic token selection individually for each video frame during training. Such token selection regularizes the training of \gls{vvit}, improves its interpretability, and is achieved by differentiable sampling of categoricals using the Gumbel-Softmax distribution. Our experiments on EchoNet-Dynamic demonstrate a consistent and notable regularization effect. The \gls{gvit} model outperforms both a random selection baseline and standard \gls{vvit}. % using multiple evaluation metrics. The \gls{gvit} is also compared against recent works on EchoNet-Dynamic where it exhibits state-of-the-art performance among end-to-end learned methods. Finally, we explore model explainability by visualizing selected patches, providing insights into how the \gls{gvit} utilizes regions known to be crucial for \gls{lvef} prediction for humans. This proposed approach, therefore, extends beyond regularization, offering enhanced interpretability for \gls{vit}s.
BibTeX
@inproceedings{nilsson2024regularizing,
  author = {Alfred Nilsson and Hossein Azizpour},
  title = {{Regularizing and Interpreting Vision Transformer by Patch Selection on Echocardiography Data}},
  booktitle = {CHIL},
  year = {2024},
  url = {https://openreview.net/forum?id=BGO91YR2c5},
}
Figure