Unsupervised brain imaging 3D anomaly detection and segmentation with transformers

Walter H. L. Pinaya, Petru-Daniel Tudosiu, Robert J. Gray, Geraint Rees, Parashkev Nachev, Sébastien Ourselin, M. Jorge Cardoso

TL;DR

This paper introduces a novel transformer-based approach for unsupervised anomaly detection and segmentation in brain imaging. By combining vector-quantized variational autoencoders (VQ-VAEs) with autoregressive transformers, the method effectively identifies and localizes abnormalities without requiring labeled pathological data, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Detecting and segmenting anomalies (like lesions) in brain imaging is crucial for diagnosis, prognosis, and treatment planning. However, these anomalies are highly diverse, making it difficult to define them by specific features. Traditional supervised methods require extensive labeled data, which is expensive and time-consuming to obtain, especially for rare conditions. Unsupervised anomaly detection aims to overcome this by learning a model of "normal" brain anatomy and identifying deviations. Existing unsupervised methods, often based on reconstruction, struggle with the fidelity of reconstructions and can misinterpret fine details as anomalies. This paper addresses the need for a robust, unsupervised method that can accurately detect and segment anomalies in brain images, even with limited computational resources and data.

Methods

The proposed method leverages two key components:

  • Vector Quantized Variational Autoencoder (VQ-VAE): This component learns a compressed, discrete latent representation of the brain images. The VQ-VAE encoder maps input images to a latent space, where each latent vector is quantized to its nearest codebook entry. The decoder then reconstructs the image from these quantized latent representations. This process effectively reduces the dimensionality and sequence length of the data, making it suitable for transformer processing.
  • Autoregressive Transformer: An ensemble of transformers is trained to model the probability distribution of the discrete latent representations learned by the VQ-VAE. By treating the latent representations as a sequence, the transformer learns to predict the likelihood of each element given its predecessors. This allows for the estimation of the probability of observing a particular latent representation in a healthy brain.

Anomaly detection and segmentation are achieved through a two-step process:

  1. Anomaly Segmentation: The transformer's predicted likelihoods for each latent variable are used to identify potentially anomalous regions. A "resampling mask" is created based on a threshold of low likelihood values. These masked latent variables are then "healed" by sampling new values from the transformer's learned distribution, generating a reconstruction without anomalies. The difference between the original and healed reconstruction highlights the anomalies, which are then segmented by thresholding the residuals.
  2. Spatial Information Integration: To mitigate the impact of blurry reconstructions from the VQ-VAE, the spatial information from the resampling mask is used to refine the anomaly segmentation. The mask is upscaled, smoothed, and multiplied with the residual maps, effectively "cleaning" areas where residuals might be high due to reconstruction limitations rather than actual anomalies.

The authors also explore different ordering strategies for the latent representation sequence fed to the transformer, finding that an ensemble of transformers with varied orderings (e.g., raster, S-curve, Hilbert) improves robustness and performance.

Datasets

The study utilizes several datasets for training and evaluation:

  • Synthetic Data:
    • MedNIST Dataset (HeadCT category): Used for training the VQ-VAE and transformers in initial experiments.
  • dSprites Dataset: Used to generate synthetic anomalies (sprites) for contaminating HeadCT images for evaluation.
  • Real Neuroimaging Data:
    • UK Biobank (UKB): A large-scale dataset used for training and evaluating on both 2D and 3D data. It includes FLAIR images and lesion masks for hyperintensities.
  • White Matter Hyperintensities Segmentation Challenge (WMH): Used for evaluating 2D anomaly segmentation.
  • Multimodal Brain Tumor Image Segmentation Benchmark (BRATS): Used for evaluating 2D anomaly segmentation.
  • Multiple Sclerosis dataset from the University Hospital of Ljubljana (MSLUB): Used for evaluating 2D anomaly segmentation.

Key Insights and Ideas

  • Transformers for Latent Space Modeling: The paper highlights the power of transformers in modeling the complex, long-range dependencies within the discrete latent representations of brain images, outperforming previous autoregressive models like PixelSNAIL.
  • VQ-VAE as an Efficient Encoder: Using VQ-VAE for dimensionality reduction is crucial for making transformer models computationally feasible for high-resolution 3D brain imaging data.
  • Leveraging Probability for Anomaly Detection: Instead of solely relying on reconstruction errors, the method uses the transformer's learned probability distribution to identify deviations from normality, leading to more specific anomaly detection.
  • The Importance of Spatial Context in Orderings: The study demonstrates that the order in which latent variables are presented to the transformer significantly impacts performance. Ensembling transformers with diverse orderings (e.g., raster, S-curve) allows the model to capture different spatial relationships and improves robustness.
  • Masked Residuals for Refined Segmentation: Integrating the spatial information from the transformer's "resampling mask" with reconstruction residuals effectively reduces false positives caused by VQ-VAE reconstruction artifacts.

Key Results

  • Synthetic Data Performance: The proposed method achieved a DICE score of 0.895 and an AUPRC of 0.956 on synthetic 2D data when using an ensemble of 8 different raster orderings, significantly outperforming other state-of-the-art methods.
  • Real 2D Neuroimaging Data: On real 2D lesion data, the method achieved a best achievable DICE score of 0.537 on the BRATS dataset and 0.429 on the WMH dataset, again outperforming baseline methods.
  • Real 3D Neuroimaging Data: For 3D anomaly segmentation, the method achieved a DICE score of 0.617 on the BRATS dataset and 0.133 on the MSLUB dataset, demonstrating its capability in higher dimensions.
  • Image-wise Anomaly Detection: The method achieved an AUROC of 0.921 for near out-of-distribution (OOD) images and 1.000 for far OOD images on synthetic data, and an AUROC of 0.866 for detecting multiple sclerosis patients in real 3D neuroimaging data when combining log-likelihood and lesion segmentation.

Limitations and Future Work

  • VQ-VAE Influence: The authors acknowledge that the VQ-VAE's architecture and training can still influence anomaly detection performance. They suggest that a more general-purpose VQ-VAE, trained on diverse data, might further improve results by reducing its ability to "heal" anomalies during encoding.
  • Computational Cost: While transformers are made feasible by VQ-VAE, they still require significant computational resources, especially for 3D data. The use of efficient transformer variants (like Performer) is a step towards mitigating this.
  • Generalizability of 3D Models: The paper notes that 3D models are generally more challenging to generalize well compared to their 2D counterparts, suggesting a need for advanced regularization strategies for 3D architectures.
  • Future Directions: The authors express optimism about the potential of transformers for anomaly detection in medical imaging and suggest exploring new network designs, incorporating a wider variety of conditioning information, and applying the approach to other medical data modalities.

Summary updated

BibTeX
@article{pinaya2022unsupervised,
  author = {Walter H. L. Pinaya and Petru-Daniel Tudosiu and Robert J. Gray and Geraint Rees and Parashkev Nachev and Sébastien Ourselin and M. Jorge Cardoso},
  title = {{Unsupervised brain imaging 3D anomaly detection and segmentation with transformers}},
  journal = {Medical Image Analysis},
  year = {2022},
  doi = {10.1016/j.media.2022.102475},
}
Figure