PSTNet: Enhanced Polyp Segmentation With Multi-Scale Alignment and Frequency Domain Integration

Wenhao Xu, Rongtao Xu, Changwei Wang, Xiuli Li, Shibiao Xu, Li Guo

Abstract

Accurate polyp segmentation in colonoscopy images is a critical step in the early detection and prevention of colorectal cancer (CRC). However, existing computer vision models often struggle with the "camouflage effect," where polyps blend into surrounding healthy tissue, and "feature misalignment," where multi-scale information fusion results in blurred boundaries. This paper introduces PSTNet, a novel architecture that leverages a Shunted Transformer backbone, frequency-domain analysis via Discrete Cosine Transforms (DCT), and a specialized feature alignment module. Experimental results across five benchmark datasets—Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS, and CVC-300—demonstrate that PSTNet achieves state-of-the-art performance, particularly in challenging low-contrast and small-polyp scenarios.

Introduction

Colorectal cancer remains a leading cause of cancer-related mortality worldwide. While colonoscopies are the gold standard for identifying and removing precancerous polyps, the visual complexity of the gastrointestinal tract presents significant challenges for both clinicians and automated systems. Two primary bottlenecks hinder current segmentation models:

  1. The Camouflage Effect: Polyps frequently exhibit color and texture characteristics nearly identical to the surrounding mucosa, making them difficult to distinguish in the standard RGB color space.
  2. Feature Misalignment: Modern architectures often fuse features from different network depths to capture both global context and local detail. However, the operations of downsampling and upsampling often introduce "pixel shifts," leading to noisy feature maps and imprecise boundaries.

PSTNet addresses these issues by integrating frequency-domain cues and a dedicated alignment strategy to refine the segmentation process.

Methodology

The PSTNet architecture is composed of four primary innovative components designed to enhance feature extraction and fusion.

Shunted Transformer Encoder

Unlike standard Transformers that process tokens at a fixed scale, the Shunted Transformer backbone utilizes "shunted self-attention." This allows the model to simultaneously model objects of varying sizes, providing a more efficient way to capture the fine-grained details necessary for identifying small or irregularly shaped polyps.

Frequency Characterization Attention Module (FCAM)

To overcome the camouflage effect, the FCAM incorporates a 2D Discrete Cosine Transform (DCT). By analyzing images in the frequency domain, the model can identify subtle structural patterns and rate-of-change signatures in pixel intensity that are often invisible in the RGB space. This "hidden" cue allows the network to distinguish polyps from healthy tissue even when their colors are nearly identical.

Feature Supplementary Alignment Module (FSAM)

The FSAM is designed to resolve the problem of feature misalignment. It employs a multi-scale subtraction strategy to highlight the unique "residual" information between adjacent layers, effectively reducing redundancy. Furthermore, it utilizes Deformable Convolutions (DCNv2) to physically align features from different scales before they are fused, resulting in significantly sharper edge detection.

Cross Perception Localization Module (CPM)

The CPM serves as the final integration stage, acting as a bridge that merges frequency-rich low-level features with high-level semantic information. This ensures that the final segmentation mask is both contextually accurate and spatially precise.

Experimental Results

PSTNet was evaluated against five major benchmarks, consistently outperforming current state-of-the-art (SOTA) models such as SegT and ADSNet.

Quantitative Performance

On the Kvasir-SEG dataset, PSTNet achieved a 93.5% mDice score and an 89.5% mIoU. These metrics indicate a high degree of overlap between the predicted masks and the ground truth.

Generalization and Robustness

The model's robustness was particularly evident on the ETIS dataset, which is widely regarded as the most difficult benchmark due to the prevalence of very small polyps. PSTNet outperformed the next best model (SegT) by 1.0% in mDice, suggesting superior generalization capabilities on unseen or difficult clinical data.

Qualitative Analysis

Visual comparisons indicate that PSTNet excels at identifying tiny polyps that other models frequently overlook. Additionally, the boundaries produced by PSTNet are cleaner and more anatomically accurate, even in low-contrast scenarios where the polyp is heavily camouflaged.

Discussion and Conclusion

PSTNet represents a significant advancement in medical image segmentation by demonstrating that frequency-domain information is a powerful supplement to traditional spatial features. By combining the Shunted Transformer's efficiency with the FSAM's alignment precision, the model effectively mitigates the most common failure modes in polyp detection.

Future Work

Future research will focus on the clinical translation of PSTNet. This includes optimizing the model for real-time performance on standard hospital hardware to support live streaming colonoscopies. Additionally, the authors aim to collaborate with medical experts to validate the model's performance across more diverse, large-scale datasets containing rare polyp morphologies.

Summary updated

Abstract

Accurate segmentation of colorectal polyps in colonoscopy images is crucial for effective diagnosis and management of colorectal cancer (CRC). However, current deep learning-based methods primarily rely on fusing RGB information across multiple scales, leading to limitations in accurately identifying polyps due to restricted RGB domain information and challenges in feature misalignment during multi-scale aggregation. To address these limitations, we propose the Polyp Segmentation Network with Shunted Transformer (PSTNet), a novel approach that integrates both RGB and frequency domain cues present in the images. PSTNet comprises three key modules: the Frequency Characterization Attention Module (FCAM) for extracting frequency cues and capturing polyp characteristics, the Feature Supplementary Alignment Module (FSAM) for aligning semantic information and reducing misalignment noise, and the Cross Perception localization Module (CPM) for synergizing frequency cues with high-level semantics to achieve efficient polyp segmentation. Extensive experiments on challenging datasets demonstrate PSTNet's significant improvement in polyp segmentation accuracy across various metrics, consistently outperforming state-of-the-art methods. The integration of frequency domain cues and the novel architectural design of PSTNet contribute to advancing computer-assisted polyp segmentation, facilitating more accurate diagnosis and management of CRC.
BibTeX
@article{xu2024pstnet,
  author = {Wenhao Xu and Rongtao Xu and Changwei Wang and Xiuli Li and Shibiao Xu and Li Guo},
  title = {{PSTNet: Enhanced Polyp Segmentation With Multi-Scale Alignment and Frequency Domain Integration}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2024},
  url = {https://openreview.net/forum?id=ZjEGLhQill},
}
Figure