Rethinking Few-Shot Medical Segmentation: A Vector Quantization View

Shiqi Huang, Tingfa Xu, Ning Shen, Feng Mu, Jianan Li

Abstract

Few-shot medical image segmentation (FSS) is challenged by the scarcity of labeled data and the inherent complexity of anatomical structures. Traditional prototype-based methods often struggle to balance local detail with global context, frequently failing to adapt to unseen tasks at test time. This paper introduces a novel perspective by framing prototype generation as a Vector Quantization (VQ) problem. We propose a framework comprising three core components: Grid-Format VQ (GFVQ), Self-Organized VQ (SOVQ), and Residual Oriented VQ (ROVQ). These mechanisms collectively enhance local detail representation, enable adaptive feature clustering, and facilitate parameter-free task adaptation. Our approach achieves state-of-the-art performance across multiple MRI benchmarks, demonstrating significant improvements in segmentation accuracy and generalizability.

Introduction

Medical image segmentation is a cornerstone of modern clinical diagnosis, yet the reliance of deep learning models on massive, manually labeled datasets remains a significant bottleneck. Few-shot segmentation (FSS) addresses this by attempting to segment new organs or lesions using only a handful of labeled examples (the "support" set).

Existing FSS methods typically extract "prototypes"—representative feature vectors—from support images to guide the segmentation of query images. However, these methods face a persistent "Goldilocks" problem: a small number of prototypes cannot capture complex anatomical nuances, while an excessive number introduces noise and misclassification. Furthermore, most models lack the flexibility to adapt to the specific, often noisy characteristics of unseen medical tasks encountered during inference.

Methodology: The VQ Framework

We reimagine prototype generation as a feature space partitioning problem using Vector Quantization. By treating prototypes as "codebook" vectors that define decision boundaries in a high-dimensional space, we can more effectively represent the underlying data distribution. Our framework introduces three specialized VQ modules:

  1. Grid-Format VQ (GFVQ): Serving as a baseline for local detail, GFVQ divides the feature map into a rigid grid and averages features within each cell. This ensures a uniform quantization across the entire spatial extent, preserving basic structural information.
  2. Self-Organized VQ (SOVQ): Inspired by Self-Organizing Maps (SOM), SOVQ adaptively clusters feature points. Unlike the rigid grid of GFVQ, SOVQ allows "neurons" to compete to represent clusters of features. This adaptive clustering provides a "global view" of the feature distribution, enabling the model to better capture irregular shapes and sharp organ boundaries.
  3. Residual Oriented VQ (ROVQ): To address the adaptation gap, ROVQ fine-tunes learned prototypes using residual information from the support image. This is a parameter-free process occurring during the forward pass, allowing the model to "adjust" to specific task characteristics without requiring additional training or backpropagation.
Comparison of Vector Quantization strategies: (a) Basic VQ, (b) Grid-Format VQ (GFVQ), (c) Self-Organized VQ (SOVQ), and (d) Residual Oriented VQ (ROVQ).

Experimental Evaluation

The proposed framework was evaluated on three diverse MRI datasets: Abdomen MRI (CHAOS challenge), Cardiac MRI (MSCMR challenge), and Prostate MRI. These datasets represent a wide range of anatomical structures and imaging challenges.

Key Results

Our method achieved state-of-the-art (SOTA) results across all benchmarks, with performance gains becoming even more significant when combined with Self-Supervised Learning (SSL) pre-training:

  • Abdomen MRI: Achieved a mean Dice score of 84.56, outperforming the previous best (SSL-ALP) by nearly 6 percentage points.
  • Cardiac MRI: Reached a mean Dice score of 84.86, a substantial improvement over the previous SOTA of 76.90.
  • Prostate MRI: Achieved a mean Dice score of 63.22, demonstrating robustness even on highly irregular and challenging structures.

Qualitative analysis shows that the dual-structure approach (GFVQ + SOVQ) effectively balances local structure with adaptive shape fitting, significantly reducing false positives compared to standard grid-based methods.

Discussion and Conclusion

This work demonstrates that shifting the perspective of prototypes from simple averages to support vectors in a VQ framework significantly enhances few-shot medical segmentation. While the iterative nature of SOVQ adds some computational overhead during inference, the gains in boundary precision and task adaptation are substantial.

Future research will explore integrating the self-organizing mechanism directly into the feature extraction backbone. This could potentially enable even more powerful unsupervised representation learning, further reducing the dependency on labeled data in clinical environments.

Summary updated

Abstract

The existing few-shot medical segmentation networks share the same practice that the more prototypes, the better performance. This phenomenon can be theoretically interpreted in Vector Quantization (VQ) view: the more prototypes, the more clusters are separated from pixel-wise feature points distributed over the full space. However, as we further think about few-shot segmentation with this perspective, it is found that the clusterization of feature points and the adaptation to unseen tasks have not received enough attention. Motivated by the observation, we propose a learning VQ mechanism consisting of grid-format VQ (GFVQ), self-organized VQ (SOVQ) and residual oriented VQ (ROVQ). To be specific, GFVQ generates the prototype matrix by averaging square grids over the spatial extent, which uniformly quantizes the local details; SOVQ adaptively assigns the feature points to different local classes and creates a new representation space where the learnable local prototypes are updated with a global view; ROVQ introduces residual information to fine-tune the aforementioned learned local prototypes without retraining, which benefits the generalization performance for the irrelevance to the training task. We empirically show that our VQ framework yields the state-of-the-art performance over abdomen, cardiac and prostate MRI datasets and expect this work will provoke a rethink of the current few-shot medical segmentation model design. Our code will soon be publicly available.
BibTeX
@inproceedings{huang2023rethinking,
  author = {Shiqi Huang and Tingfa Xu and Ning Shen and Feng Mu and Jianan Li},
  title = {{Rethinking Few-Shot Medical Segmentation: A Vector Quantization View}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00300},
}
Figure