Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection

Ashay Patel, Petru-Daniel Tudosiu, Walter H. L. Pinaya, Gary J. Cook, Vicky Goh, Sébastien Ourselin, M. Jorge Cardoso

TL;DR

This paper introduces a novel multi-modal approach for unsupervised anomaly detection in whole-body PET scans, leveraging cross-attention transformers to integrate anatomical information from CT scans. This method significantly improves cancer detection accuracy, especially when healthy training data is scarce, and introduces a more robust kernel density estimation technique for anomaly mapping.

What Problem Does This Paper Address?

Cancer's heterogeneity makes it challenging to develop universal, supervised models for detection. Unsupervised anomaly detection, which learns "normal" tissue patterns and flags deviations, offers a promising alternative. However, accurately identifying subtle anomalies in whole-body PET scans requires models that can grasp complex, long-range relationships between organs and their imaging characteristics. Existing transformer-based anomaly detection methods, while powerful, often rely solely on PET data. This paper addresses the need to incorporate complementary anatomical information from CT scans to enhance the precision and robustness of these anomaly detection systems, particularly in scenarios where obtaining purely healthy PET data for training is difficult or impossible.

Methods

The core of the proposed method is a transformer-based architecture enhanced with cross-attention for multi-modal integration.

  • VQ-GAN (Vector Quantized Variational Autoencoder with Generative Adversarial Network): This component first encodes the 3D PET (and CT) images into a discrete latent space. This process is crucial for making the data amenable to transformer processing, which typically works with sequential or categorical data. The VQ-GAN itself is a combination of a VQ-VAE and a GAN, aiming for high-fidelity reconstructions.
  • Transformer with Cross-Attention: The discrete latent representations from the VQ-GAN are then fed into a transformer. The key innovation here is the cross-attention mechanism. This allows the transformer to attend not only to the PET sequence (self-attention) but also to the corresponding CT latent sequence. This effectively injects anatomical context from the CT scan into the PET anomaly detection process at each layer of the transformer.
  • Kernel Density Estimation (KDE) for Anomaly Mapping: Instead of relying on simple residual maps (difference between original and reconstructed images), the authors propose using KDE. This method accounts for the often non-Gaussian and multi-modal nature of uptake patterns and model uncertainty, leading to more robust anomaly maps. Stochasticity is introduced via dropout in the VQ-GAN decoder and resampling from the transformer to generate multiple reconstructions, upon which KDE is fitted.

Datasets

The study utilizes a combination of datasets for training, validation, and testing:

  • Private Dataset: 83 co-registered 18F-FDG PET/CT images with various cancer types. Used for training, validation, and testing.
  • NSCLC Radiogenomics Dataset: 211 co-registered 18F-FDG PET/CT samples of Non-small cell lung cancer. Used for training, validation, and testing.
  • AutoPET Dataset: A large dataset of 1014 PET scans, including 430 non-anomalous scans. Used to create a separate training dataset consisting of only normal cases.
  • CPTAC Dataset: 14 co-registered whole-body 18F-FDG PET/CT images. Used for fully out-of-sample testing to assess generalization.

Key Insights and Ideas

  • CT Conditioning is Crucial: Integrating anatomical information from CT scans via cross-attention significantly boosts anomaly detection performance, especially in complex regions where physiological uptake can mimic pathological findings.
  • Codebook Size Matters: The choice of codebook size in the VQ-GAN architecture has a substantial impact on downstream anomaly detection performance, highlighting the importance of this hyperparameter beyond just reconstruction quality.
  • KDE Outperforms Residual Maps: The proposed Kernel Density Estimation approach for generating anomaly maps is more robust than traditional residual maps, particularly in handling multi-modal distributions and model uncertainty, leading to better differentiation of physiological vs. pathological uptake.
  • Robustness to Unseen Data: The model demonstrates strong generalization capabilities, performing well on out-of-sample datasets (CPTAC) and even when trained on data containing anomalies, which is a significant advantage in medical imaging where purely healthy datasets are rare.

Key Results

The proposed multi-modal transformer with KDE achieved state-of-the-art results:

  • Hold-out Testing: The VQ-GAN + Transformer + CT conditioning + KDE model achieved a DICE score of 0.562 and an AUPRC of 0.561 on the hold-out testing data, significantly outperforming baseline autoencoder and F-AnoGAN methods.
  • Out-of-Sample Testing (CPTAC): The same model achieved a DICE score of 0.610 and an AUPRC of 0.604 on the CPTAC dataset, demonstrating excellent generalization.
  • Ablation Studies: CT conditioning alone provided a statistically significant improvement in DICE and AUPRC. Incorporating KDE further boosted performance, especially in AUPRC.
  • Codebook Optimization: Experiments showed that a codebook vocabulary size of 256 with a dimension of 128 yielded the best results for anomaly detection.

Limitations and Future Work

  • False Positives: Despite improvements, the model still exhibits some false positives, particularly in areas with high physiological uptake (e.g., injection sites, amputated limbs) or due to pre-existing patient conditions.
  • Data Scarcity: While the method is robust to limited healthy training data, increasing the amount of diverse training data could further improve performance.
  • Future Directions: The authors suggest exploring weak or self-supervision techniques to mitigate outliers and incorporating model ensembling at different downsampling levels to capture both local and global context more effectively. Further investigation into more sophisticated methods for differentiating physiological from pathological uptake is also warranted.

Summary updated

Abstract

Cancer is a highly heterogeneous condition that can occur almost anywhere in the human body. 18F-fluorodeoxyglucose is an imaging modality commonly used to detect cancer due to its high sensitivity and clear visualisation of the pattern of metabolic activity. Nonetheless, as cancer is highly heterogeneous, it is challenging to train general-purpose discriminative cancer detection models, with data availability and disease complexity often cited as a limiting factor. Unsupervised anomaly detection models have been suggested as a putative solution. These models learn a healthy representation of tissue and detect cancer by predicting deviations from the healthy norm, which requires models capable of accurately learning long-range interactions between organs and their imaging patterns with high levels of expressivity. Such characteristics are suitably satisfied by transformers, which have been shown to generate state-of-the-art results in unsupervised anomaly detection by training on normal data. This work expands upon such approaches by introducing multi-modal conditioning of the transformer via cross-attention i.e. supplying anatomical reference from paired CT. Using 294 whole-body PET/CT samples, we show that our anomaly detection method is robust and capable of achieving accurate cancer localization results even in cases where normal training data is unavailable. In addition, we show the efficacy of this approach on out-of-sample data showcasing the generalizability of this approach with limited training data. Lastly, we propose to combine model uncertainty with a new kernel density estimation approach, and show that it provides clinically and statistically significant improvements when compared to the classic residual-based anomaly maps. Overall, a superior performance is demonstrated against leading state-of-the-art alternatives, drawing attention to the potential of these approaches.
BibTeX
@inproceedings{patel2022cross,
  author = {Ashay Patel and Petru-Daniel Tudosiu and Walter H. L. Pinaya and Gary J. Cook and Vicky Goh and Sébastien Ourselin and M. Jorge Cardoso},
  title = {{Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection}},
  booktitle = {DGM4MICCAI @ MICCAI},
  year = {2022},
  doi = {10.1007/978-3-031-18576-2\_2},
}
Figure