Path-SAM2: Transfer SAM2 for digital pathology semantic segmentation

Mingya Zhang, Liang Wang, Zhihao Chen, Yiyuan Ge, Xianping Tao

TL;DR

Path-SAM2 is the first framework to adapt the Segment Anything Model 2 (SAM2) for digital pathology by integrating a domain-specific encoder (UNI) and a learnable Kolmogorov-Arnold Network (KAN) prompt module. This approach eliminates the need for manual prompting while achieving state-of-the-art (SOTA) performance in segmenting complex tissue structures.

What Problem Does This Paper Address?

Semantic segmentation in digital pathology—identifying specific tissue structures, cell types, or lesions—is critical for disease diagnosis and prognosis. While foundation models like the original Segment Anything Model (SAM) and its successor SAM2 have revolutionized general computer vision, they face two major hurdles in pathology:

  1. Domain Gap: These models are primarily trained on natural images (dogs, cars, trees) and lack the specialized "prior knowledge" required to distinguish subtle pathological features in H&E-stained slides.
  2. Prompt Dependency: SAM-based models typically require manual prompts (like clicking on an object or drawing a box) to work effectively. In a high-throughput clinical environment, requiring a pathologist to manually prompt every image is impractical and leads to inconsistent results.

Methods

The authors introduce Path-SAM2, a pipeline designed to make SAM2 "pathology-aware" and fully automated. The architecture consists of several key innovations:

  • Dual-Encoder Strategy: Path-SAM2 uses two encoders in parallel. It retains the original SAM2 image encoder (Hiera) but supplements it with the UNI encoder, a cutting-edge self-supervised model pre-trained on over 100 million pathology images. The features from both are concatenated to provide a rich representation of the image.
  • KAN Classification Module: To replace manual prompts, the authors utilize Kolmogorov-Arnold Networks (KANs). Unlike traditional Multi-Layer Perceptrons (MLPs) that use fixed activation functions on nodes, KANs use learnable activation functions on the "edges" (connections). This module generates $k$ learnable prompt tokens (one for each tissue category), allowing the model to perform semantic segmentation automatically.
  • Hybrid Decoding: The fused features and KAN-generated prompts are fed into a hybrid mask decoder that outputs both the segmentation masks and an Intersection over Union (IoU) prediction for each class.

Datasets

The model was evaluated on three widely recognized pathology benchmarks:

  • EBHI: 4,456 H&E-stained images covering six distinct histological categories.
  • CRAG (Colorectal Adenocarcinoma Gland): 213 high-resolution images focused on gland segmentation.
  • GlaS (Gland Segmentation): 165 images from colorectal adenocarcinoma sections.

Key Insights and Ideas

  • Pathology-Specific Priors are Essential: The paper demonstrates that even the advanced SAM2 architecture performs poorly on pathology tasks (often failing to reach 50% IoU) without the domain-specific knowledge provided by the UNI encoder.
  • Automating the "Human in the Loop": By training the KAN module to produce prompt tokens, the researchers successfully converted a prompt-based interactive tool into a fully automated semantic segmentation engine.
  • The Power of KANs: The authors highlight that KANs offer better interpretability and precision for classification tasks compared to standard MLPs, which directly translates to more accurate prompt generation for the decoder.

Key Results

Path-SAM2 achieved SOTA results across all tested datasets, significantly outperforming vanilla SAM, SAM2, and even medical-specific versions like MedSAM2.

  • EBHI Dataset: Path-SAM2 reached a Dice Similarity Coefficient (DSC) of 92.23 and an IoU of 93.17. For comparison, vanilla SAM2 with manual prompts only achieved an IoU of 69.25.
  • GlaS Dataset: The model achieved a 92.02 IoU, a massive leap over the 67.68 IoU of prompted SAM2.
  • Ablation Study: Removing the KAN module (the "w.o. KAN" test) caused a significant drop in performance (e.g., DSC on GlaS dropped from 85.3 to 71.45), proving that the learnable prompt mechanism is a core driver of the model's success.

Limitations and Future Work

While Path-SAM2 shows impressive results, the authors note a few areas for further exploration:

  • Computational Efficiency: Although KANs are parameter-efficient, the dual-encoder setup (SAM2 + UNI) increases the computational footprint during inference compared to single-encoder models.
  • Video Potential: SAM2 was designed for video segmentation. While this paper focuses on static images, the authors hint that the Path-SAM2 framework could eventually be extended to analyze pathology videos or multi-focal plane scans.

The code and model weights are expected to be released at: https://github.com/simzhangbest/SAM2PATH

Summary updated

Abstract

The semantic segmentation task in pathology plays an indispensable role in assisting physicians in determining the condition of tissue lesions. With the proposal of Segment Anything Model (SAM), more and more foundation models have seen rapid development in the field of image segmentation. Recently, SAM2 has garnered widespread attention in both natural image and medical image segmentation. Compared to SAM, it has significantly improved in terms of segmentation accuracy and generalization performance. We compared the foundational models based on SAM and found that their performance in semantic segmentation of pathological images was hardly satisfactory. In this paper, we propose Path-SAM2, which for the first time adapts the SAM2 model to cater to the task of pathological semantic segmentation. We integrate the largest pretrained vision encoder for histopathology (UNI) with the original SAM2 encoder, adding more pathology-based prior knowledge. Additionally, we introduce a learnable Kolmogorov-Arnold Networks (KAN) classification module to replace the manual prompt process. In three adenoma pathological datasets, Path-SAM2 has achieved state-of-the-art performance.This study demonstrates the great potential of adapting SAM2 to pathology image segmentation tasks. We plan to release the code and model weights for this paper at: https://github.com/simzhangbest/SAM2PATH
BibTeX
@misc{zhang2024path,
  author = {Mingya Zhang and Liang Wang and Zhihao Chen and Yiyuan Ge and Xianping Tao},
  title = {{Path-SAM2: Transfer SAM2 for digital pathology semantic segmentation}},
  year = {2024},
  eprint = {2408.03651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.03651},
}
Figure