Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection

Ashay Patel, Petru-Daniel Tudosiu, Walter Hugo Lopez Pinaya, Gary J. Cook, Vicky Goh, Sébastien Ourselin, M. Jorge Cardoso

TL;DR

This paper introduces a novel transformer-based approach for unsupervised anomaly detection in whole-body PET scans, leveraging multi-modal CT data for improved accuracy. By incorporating cross-attention and a kernel density estimation (KDE) for uncertainty, the method achieves state-of-the-art results, even when trained on data containing anomalies, which is crucial for medical imaging where healthy datasets are scarce.

What Problem Does This Paper Address?

Detecting and segmenting cancerous anomalies in medical imaging, particularly Positron Emission Tomography (PET), is vital for cancer diagnosis, staging, and treatment planning. PET excels at visualizing metabolic activity, a key indicator of cancer. However, PET scans can be noisy and exhibit heterogeneous uptake patterns, making anomaly detection challenging, especially for small or metastatic lesions. While unsupervised anomaly detection methods, which learn from "healthy" data, have shown promise, they typically require clean training data. This is a significant limitation in oncology, where obtaining purely healthy datasets for certain imaging modalities or patient populations is difficult or unethical. Furthermore, high uptake in PET doesn't always equate to pathology (e.g., in organs like the heart or kidneys), necessitating additional context to differentiate physiological from pathological signals.

Methods

The authors propose a multi-modal, transformer-based anomaly detection pipeline. The core idea is to use transformers, known for their ability to capture long-range dependencies, to learn the probability distribution of healthy tissue patterns.

Here's a breakdown of their approach:

  • VQ-GAN for Latent Representation: To make PET images suitable for transformers (which typically process sequential data), they first use a Vector Quantized Generative Adversarial Network (VQ-GAN). This model learns a discrete latent representation of the PET images, effectively compressing them into a sequence of "tokens."
  • Transformer for Anomaly Detection: A Performer model (an efficient transformer variant) is then trained on these latent sequences. The transformer learns to predict the probability of each token appearing at a given position. Anomalies are identified as tokens with low predicted probabilities, indicating deviations from the learned "healthy" distribution.
  • CT Conditioning with Cross-Attention: This is a key innovation. To provide anatomical context and differentiate physiological from pathological uptake, they incorporate registered CT images. The CT images are also encoded into a latent sequence. The transformer is augmented with cross-attention layers, allowing it to attend to the CT latent sequence while processing the PET latent sequence. This helps the model understand that certain high-uptake patterns in PET might be explained by anatomical structures visible in CT.
  • Kernel Density Estimation (KDE) for Robust Anomaly Maps: Instead of relying on simple residual maps (difference between original and reconstructed images), which can be sensitive to noise and thresholding, they employ KDE. By introducing model uncertainty (e.g., through dropout in the VQ-GAN decoder) and generating multiple "healed" reconstructions, they estimate the probability density function of voxel intensities. This non-parametric approach is more robust to non-Gaussian error distributions observed in the data and provides a more statistically sound anomaly map.

Datasets

The study utilized 83 whole-body PET/CT samples containing various cancer types for training and evaluation. The paper does not mention the creation of a new benchmark dataset.

Key Insights and Ideas

  • Multi-modal Fusion is Crucial for Context: The integration of CT data via cross-attention significantly improves anomaly detection by providing anatomical context, helping to disambiguate physiological high-uptake regions from actual anomalies.
  • Transformers Excel at Long-Range Dependencies: The transformer architecture's ability to model long-range interactions is well-suited for understanding the complex spatial relationships within whole-body PET scans.
  • KDE Offers Robust Anomaly Scoring: Moving beyond simple residual maps to KDE-based anomaly scoring, which accounts for model uncertainty and non-Gaussian error distributions, leads to more reliable and precise anomaly localization.
  • Training on Anomalous Data is Feasible: The proposed method demonstrates robustness even when trained on datasets containing anomalies, a critical advantage for medical imaging applications where purely healthy training data is often unavailable.

Key Results

  • The proposed VQ-GAN + Transformer + CT conditioning + KDE model achieved a DICE score of 0.505 and an AUPRC of 0.501 on the test set.
  • This represents a statistically significant improvement (P<.001) over a VAE-based approach (DICE 0.359, AUPRC 0.282) and a PET-only transformer approach (DICE 0.424, AUPRC 0.301).
  • The addition of CT conditioning improved the DICE score from 0.424 to 0.468, and KDE further boosted it to 0.505.
  • Qualitative results (Figure 2) show improved precision and better differentiation of anomalies compared to baseline methods.
  • A final post-processing step of growing segmentations to meet clinical standards (40% of maximum intensity) further improved the DICE score to 0.575, though it slightly reduced AUPRC.

Limitations and Future Work

  • False Positives: The authors acknowledge ongoing difficulties in differentiating physiological uptake from pathological uptake, leading to varying cases of false positives. This may be influenced by patient-specific factors like general health, radiotracer dosage, and injection timing.
  • Future Directions:
    • Patient/Procedure Conditioning: Providing further conditioning related to patient history and procedure details could help mitigate false positives.
  • Further Investigation: The authors hope their work will inspire further research into anomaly detection with conditioned transformers using multi-modal medical imaging.

Summary updated

Abstract

Cancer is a highly heterogeneous condition that can occur almost anywhere in the human body. 18F-fluorodeoxyglucose is an imaging modality commonly used to detect cancer due to its high sensitivity and clear visualisation of the pattern of metabolic activity. Nonetheless, as cancer is highly heterogeneous, it is challenging to train general-purpose discriminative cancer detection models, with data availability and disease complexity often cited as a limiting factor. Unsupervised anomaly detection models have been suggested as a putative solution. These models learn a healthy representation of tissue and detect cancer by predicting deviations from the healthy norm, which requires models capable of accurately learning long-range interactions between organs and their imaging patterns with high levels of expressivity. Such characteristics are suitably satisfied by transformers, which have been shown to generate state-of-the-art results in unsupervised anomaly detection by training on normal data. This work expands upon such approaches by introducing multi-modal conditioning of the transformer via cross-attention i.e. supplying anatomical reference from paired CT. Using 294 whole-body PET/CT samples, we show that our anomaly detection method is robust and capable of achieving accurate cancer localization results even in cases where normal training data is unavailable. In addition, we show the efficacy of this approach on out-of-sample data showcasing the generalizability of this approach with limited training data. Lastly, we propose to combine model uncertainty with a new kernel density estimation approach, and show that it provides clinically and statistically significant improvements when compared to the classic residual-based anomaly maps. Overall, a superior performance is demonstrated against leading state-of-the-art alternatives, drawing attention to the potential of these approaches.
BibTeX
@inproceedings{patel2022cross,
  author = {Ashay Patel and Petru-Daniel Tudosiu and Walter Hugo Lopez Pinaya and Gary J. Cook and Vicky Goh and Sébastien Ourselin and M. Jorge Cardoso},
  title = {{Cross Attention Transformers for Multi-modal Unsupervised Whole-Body PET Anomaly Detection}},
  booktitle = {DGM4MICCAI @ MICCAI},
  year = {2022},
  doi = {10.1007/978-3-031-18576-2\_2},
}
Figure