Abstract
The scarcity of annotated data, particularly for rare diseases, limits the variability of training data and the range of detectable lesions, presenting a significant challenge for supervised anomaly detection in medical imaging. To solve this problem, we propose a novel unsupervised method for medical image anomaly detection: Attention-Augmented Differentiable top-k Feature Adaptation (ADFA). The method utilizes Wide-ResNet50-2 (WR50) network pre-trained on ImageNet to extract initial feature representations. To reduce the channel dimensionality while preserving relevant channel information, we employ an attention-augmented patch descriptor on the extracted features. We then apply differentiable top-k feature adaptation to train the patch descriptor, mapping the extracted feature representations to a new vector space, enabling effective detection of anomalies. Experiments show that ADFA outperforms state-of-the-art (SOTA) methods on multiple challenging medical image datasets, confirming its effectiveness in medical anomaly detection.
Introduction
Detecting anomalies in medical images—such as tumors in an MRI or lesions in an ultrasound—is a high-stakes task. While supervised deep learning has made massive strides, it faces significant hurdles when applied to rare diseases. Training a model to find a "needle in a haystack" is difficult when examples of the "needle" are scarce. Furthermore, expert radiologists' time for labeling is expensive and limited.
Unsupervised methods offer a solution by learning only from "normal" images. However, existing approaches like reconstruction-based models or standard embedding-similarity methods often struggle with the high variability and feature redundancy inherent in medical imaging. ADFA addresses these challenges by learning the intricate patterns of healthy tissue and mapping features into a space where anomalies stand out clearly.
Methodology
The ADFA framework consists of three main stages designed to refine raw features into discriminative representations for anomaly detection.
1. Feature Extraction
The process begins with a pre-trained Wide-ResNet50-2 (WR50) backbone. To incorporate spatial context, 2D coordinate channels are concatenated with the extracted feature maps, providing the model with positional information crucial for identifying localized abnormalities.
2. Attention-Augmented Patch Descriptor
Standard pre-trained models often produce redundant features. ADFA employs a custom attention module using both Global Max Pooling (GMP) and Global Average Pooling (GAP) to highlight the most informative channels. This reduces dimensionality while preserving critical diagnostic information, effectively filtering out "noise" that isn't relevant to medical pathology.
3. Differentiable Top-k Adaptation
The core innovation of ADFA is the differentiable top-k operator. Instead of a single global average for "normal," ADFA maintains a center feature bank. During training, a "soft" top-k search identifies the $K$ nearest normal centers. This allows the model to be optimized via gradient descent, pulling normal features closer to their respective neighbors in a new vector space. This multi-center approach prevents the model from being confused by the natural diversity of healthy anatomy (e.g., bone vs. soft tissue).
Experiments and Results
The researchers validated ADFA on four distinct medical imaging benchmarks: BrainMRI (tumors), Covid (chest X-rays), BUSI (breast ultrasound), and SIPaKMeD (cervical pap smears).
ADFA consistently outperformed or matched state-of-the-art (SOTA) models such as PaDiM, SPADE, and Patch SVDD. It achieved the highest AUROC on the BUSI (0.966) and SIPaKMeD (0.972) datasets. Notably, on the SIPaKMeD dataset, ADFA improved the AUROC by 3.7% over the previous best method. Ablation studies confirmed that the attention module and the differentiable top-k adaptation were both vital to these performance gains.
Conclusion
ADFA provides a robust unsupervised framework for medical anomaly detection by refining how AI "looks" at normal anatomy. By combining channel attention with a differentiable multi-center adaptation strategy, it effectively identifies rare pathologies without requiring extensive labeled datasets. Future work will focus on extending ADFA to anomaly localization, providing clinicians with visual "heatmaps" to pinpoint lesions, and exploring its utility in non-medical domains.
Summary updated