Prototype Learning for Explainable Brain Age Prediction

Linde S. Hesse, Nicola K. Dinsdale, Ana I. L. NambureteWACV 2024

TL;DR

This paper introduces ExPeRT, a novel explainable deep learning model for regression tasks, specifically brain age prediction. By combining prototype learning with optimal transport for patch-level matching, ExPeRT provides interpretable, example-based explanations for its predictions while achieving state-of-the-art performance.

What Problem Does This Paper Address?

Deep learning models, while powerful, often function as "black boxes," making it difficult to understand their decision-making process. This lack of transparency is a significant barrier to their adoption in critical fields like clinical practice, where trust and understanding are paramount. While prototype-based models offer inherent explainability, they have traditionally been designed for classification tasks. However, many important medical imaging tasks, such as predicting brain age, are continuous regression problems. Existing methods for explaining regression models, like saliency maps, can be noisy and unfaithful. This paper addresses the need for an explainable model that can handle regression tasks effectively, providing transparent and insightful predictions.

Methods

ExPeRT is built upon a prototype learning framework. The core idea is to learn a set of "prototypes" in a latent space, where each prototype represents a typical example of a whole training image. For regression, these prototypes are assigned continuous labels (e.g., age).

The key innovations in ExPeRT are:

  • Prototype Projection: Prototypes are constrained to be actual learned representations of training images, allowing for direct visualization and interpretation.
  • Metric Learning with Optimal Transport (OT): ExPeRT uses a novel metric loss that enforces distances between image representations and prototypes in latent space to be proportional to their differences in label space. Crucially, to incorporate spatial detail, the image-level distances are decomposed into patch-level similarities. Optimal Transport is employed to structurally match patches between an image and a prototype, creating a cost matrix that captures these patch-wise relationships. This allows for fine-grained, example-based explanations at the patch level.
  • Consistency Loss: To ensure anatomically correct matches, a consistency loss is introduced. This loss encourages similar spatial locations in an image and its transformed version to have similar latent representations, while pushing apart representations of different spatial locations. This acts as a form of unsupervised, contrastive learning.
  • Inference: Predictions are made by taking a weighted average of prototype labels, where the weights are determined by the distances to the prototypes within a certain radius.

The entire network is designed to be end-to-end differentiable, allowing for seamless training.

Datasets

The proposed ExPeRT model was evaluated on two datasets:

  • Fetal Ultrasound (US) Dataset: 2D ultrasound images from the INTERGROWTH-21st Fetal Growth Longitudinal Study were used to predict gestational age. A total of 4290 volumes were processed into 4290 2D images of size 160x160.
  • Adult MRI Dataset: T1 MRI images from the IXI dataset were used for brain age prediction. This dataset comprised 561 MR images, each of size 160x192, with ages ranging from 19 to 86.

Key Insights and Ideas

  • Explainability Doesn't Necessarily Hurt Performance: A central finding is that achieving increased explainability does not inherently lead to a decrease in prediction performance. ExPeRT achieves state-of-the-art results while being interpretable.
  • Patch-Level Detail for Regression: Decomposing image-level distances into patch-level similarities using Optimal Transport is crucial for providing detailed, spatially localized explanations in regression tasks, moving beyond simple pixel importance.
  • Unsupervised Consistency for Anatomical Alignment: The consistency loss, which uses geometric transformations (like flipping) without explicit labels, effectively enforces anatomically plausible matches between image patches and prototypes, particularly important for tasks like brain imaging.
  • Prototypes as Visualizable Examples: By constraining prototypes to be actual training image representations, the model provides intuitive, example-based explanations that are easy for humans to understand.

Key Results

  • State-of-the-Art Performance: ExPeRT achieved competitive, and in some cases state-of-the-art, prediction performance (measured by Mean Absolute Error - MAE) on both the fetal US and adult MRI datasets, outperforming several baseline models. For example, on the adult MRI dataset, ExPeRT achieved an MAE of 5.57 years, comparable to or better than other methods.
  • Improved Prediction with Add-on Layers and OT: Ablation studies showed that the add-on layers (h) and the patch-based OT matching significantly improved prediction performance compared to simpler approaches.
  • Effectiveness of Consistency Loss: For the US dataset, the consistency loss was shown to be critical for correctly matching hemispheres, demonstrating its importance in tasks where anatomical symmetry is relevant.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the model's hyperparameters were found to be generally stable, the inference radius (r) was noted to have a significant impact on performance, though it can be optimized offline.
  • Geometric Transformations: The consistency loss currently uses simple geometric transformations like flipping. The authors suggest that future work could explore other transformations based on the geometric variations present in the dataset.
  • Generalizability: While the model is versatile and can be applied to other continuous regression problems, its performance on a wider range of medical imaging tasks and other domains would be valuable to explore.
  • No Hemisphere Labels Used: The consistency loss operates in an unsupervised manner, not requiring explicit hemisphere labels, which is a strength. However, exploring how explicit anatomical labels might further enhance performance or explainability could be a future direction.

Summary updated

BibTeX
@inproceedings{hesse2024prototype,
  author = {Linde S. Hesse and Nicola K. Dinsdale and Ana I. L. Namburete},
  title = {{Prototype Learning for Explainable Brain Age Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure