Anomaly Detection-Inspired Few-Shot Medical Image Segmentation Through Self-Supervision With Supervoxels

Stine Hansen, Srishti Gautam, Robert Jenssen, Michael Kampffmeyer

Abstract

Recent work has shown that label-efficient few-shot learning through self-supervision can achieve promising medical image segmentation results. However, few-shot segmentation models typically rely on prototype representations of the semantic classes, resulting in a loss of local information that can degrade performance. This is particularly problematic for the typically large and highly heterogeneous background class in medical image segmentation. Previous works have attempted to address this issue by learning additional prototypes for each class, but since these prototypes are based on a limited number of slices, this ad-hoc solution is often insufficient to capture background properties. Motivated by the observation that the foreground class (e.g., a specific organ) is relatively homogeneous, we propose a novel anomaly detection-inspired approach to few-shot medical image segmentation. Our model, ADNet, refrains from modeling the background explicitly, relying instead on a single foreground prototype to compute anomaly scores for all query pixels. The segmentation is then performed by thresholding these scores using a learned threshold. Assisted by a novel self-supervision task that exploits the 3D structure of medical images through supervoxels (vSSL), our proposed model outperforms previous state-of-the-art approaches on representative MRI datasets for abdominal and cardiac segmentation.

Introduction

Deep learning has revolutionized medical image segmentation, yet its reliance on massive, manually annotated datasets remains a significant bottleneck. Few-Shot Segmentation (FSS) has emerged as a solution, enabling models to segment new organs using only a handful of labeled examples (the "support set").

Standard FSS models typically utilize "prototypes"—feature vectors that represent specific classes. To segment a query image, the model compares each pixel's features to these prototypes. However, a persistent challenge in medical imaging is the "Heterogeneity Headache." While a target organ like the liver is relatively uniform, the "background" is a chaotic mix of various tissues, organs, and artifacts. Attempting to represent this diverse background with a few prototypes often leads to over-segmentation, where the model incorrectly labels background pixels as the target organ.

Methodology

ADNet: Anomaly Detection-Inspired Network

ADNet departs from standard FSS by treating segmentation as an anomaly detection task. The architecture is built on three core pillars:

  1. Foreground-Only Modeling: Rather than attempting to model the complex background, ADNet creates a prototype solely for the foreground (the organ of interest).
  2. Anomaly Scoring: The model calculates an "anomaly score" for every pixel in the query image based on its cosine similarity to the foreground prototype. Pixels that are dissimilar to the prototype receive high anomaly scores.
  3. Learned Thresholding: Instead of a simple nearest-neighbor classification, ADNet employs a learned threshold ($T$). If a pixel's similarity to the foreground prototype falls below this threshold, it is classified as background (an anomaly). This mechanism acts as a safety valve, significantly reducing false positives.

vSSL: Supervoxel Self-Supervision

To train the model without manual labels, we introduce a novel self-supervision task called vSSL. While previous methods used 2D superpixels, vSSL utilizes supervoxels—3D clusters of similar pixels within a volumetric scan. By selecting a random supervoxel to act as a "pseudo-organ" and sampling different 2D slices from the 3D volume for the support and query sets, the model is forced to learn the 3D spatial consistency inherent in medical scans. This teaches the model that an organ in one slice should maintain feature consistency with the same organ in adjacent slices.

Experimental Setup

Datasets and Protocols

We evaluated ADNet on two representative MRI benchmarks:

  • MS-CMRSeg: Cardiac MRI scans focusing on the left ventricle and myocardium.
  • CHAOS: Abdominal MRI scans for segmenting the liver, kidneys, and spleen.

Crucially, we introduced a more rigorous evaluation protocol, EP2. Unlike standard protocols that only present the model with slices containing the target organ, EP2 requires the model to segment the entire volume, including slices where the organ is absent. This provides a more realistic assessment of clinical utility.

Results and Discussion

Performance Gains

Under the realistic EP2 protocol, ADNet demonstrated a substantial improvement over the previous state-of-the-art, ALPNet. On the CHAOS dataset, ADNet improved the Mean Dice score by over 20 percentage points. This suggests that ignoring the background is more effective than trying to model it poorly.

Efficiency and Robustness

ADNet achieves these results with higher efficiency; while competing models may use hundreds of background prototypes to capture heterogeneity, ADNet uses zero. Furthermore, ablation studies indicated that the learned threshold and the foreground-only approach significantly reduced false positives in difficult regions, such as the myocardium.

3D Context

Experiments with a 3D ResNeXt-101 backbone showed that incorporating volumetric context further improves the separation between adjacent, similar-looking organs, such as the left kidney and the spleen. This highlights the potential of moving beyond 2D-only processing in few-shot medical imaging.

Conclusion

ADNet demonstrates that in the context of few-shot medical image segmentation, "less is more." By focusing entirely on the homogeneity of the foreground and treating the background as an anomaly, we can bypass the difficulties of modeling complex anatomical environments. Combined with 3D supervoxel self-supervision, this approach sets a new benchmark for label-efficient medical image analysis, particularly in realistic scenarios where the target organ's location is not known a priori.

Summary updated

Abstract

Recent work has shown that label-efficient few-shot learning through self-supervision can achieve promising medical image segmentation results. However, few-shot segmentation models typically rely on prototype representations of the semantic classes, resulting in a loss of local information that can degrade performance. This is particularly problematic for the typically large and highly heterogeneous background class in medical image segmentation problems. Previous works have attempted to address this issue by learning additional prototypes for each class, but since the prototypes are based on a limited number of slices, we argue that this ad-hoc solution is insufficient to capture the background properties. Motivated by this, and the observation that the foreground class (e.g., one organ) is relatively homogeneous, we propose a novel anomaly detection-inspired approach to few-shot medical image segmentation in which we refrain from modeling the background explicitly. Instead, we rely solely on a single foreground prototype to compute anomaly scores for all query pixels. The segmentation is then performed by thresholding these anomaly scores using a learned threshold. Assisted by a novel self-supervision task that exploits the 3D structure of medical images through supervoxels, our proposed anomaly detection-inspired few-shot medical image segmentation model outperforms previous state-of-the-art approaches on two representative MRI datasets for the tasks of abdominal organ segmentation and cardiac segmentation.
BibTeX
@article{hansen2022anomaly,
  author = {Stine Hansen and Srishti Gautam and Robert Jenssen and Michael Kampffmeyer},
  title = {{Anomaly Detection-Inspired Few-Shot Medical Image Segmentation Through Self-Supervision With Supervoxels}},
  journal = {Medical Image Analysis},
  year = {2022},
  doi = {10.1016/j.media.2022.102385},
}
Figure