Sub-Band Based Attention for Robust Polyp Segmentation

Xianyong Fang, Yuqing Shi, Qingqing Guo, Linbo Wang, Zhengyi LiuIJCAI 2023 · DOI

TL;DR

This paper introduces SBA-Net, a novel framework for polyp segmentation in colonoscopy images. It leverages a "Sub-Band Based Attention" (SBA) module that focuses on specific frequency bands within image features, inspired by the observation that middle-frequency bands become increasingly important during CNN processing. Combined with a "Transformer Attended Convolution" (TAC) encoder, SBA-Net achieves robust polyp segmentation, particularly in challenging cases with blurry boundaries and low contrast.

What Problem Does This Paper Address?

Colorectal cancer is a significant global health concern, and early detection through colonoscopy is crucial. Accurate polyp segmentation in colonoscopy images is a key step in this process, but it's a challenging task. Polyps often have subtle appearances, blending with surrounding tissues, and can exhibit blurry boundaries and low contrast, making them difficult to distinguish. Existing deep learning methods, while improving, still struggle with these challenging visual characteristics. The paper aims to address this by developing a more robust segmentation method that can better capture the subtle features of polyps.

Methods

The core of SBA-Net lies in two novel components:

  • Sub-Band Based Attention (SBA) Module: This module is inspired by an empirical observation: as image features pass through convolutional neural network (CNN) stages, the middle-frequency sub-bands of the spectral domain representation become disproportionately stronger. The SBA module leverages this by applying attention mechanisms that focus on either the middle or high-frequency sub-bands of encoder features. This is achieved by transforming features into the frequency domain (using FFT), applying Gaussian filters to isolate specific frequency bands, and then using these to enhance decoder features. The idea is that these specific sub-bands contain crucial information for distinguishing polyps.
  • Transformer Attended Convolution (TAC) Module: To ensure the SBA module receives informative features, the paper proposes a novel encoder block called TAC. This module combines the strengths of CNNs (for local details) and Transformers (for long-range contextual information). Specifically, it uses Transformer features (from a Pyramid Vision Transformer, PVT) as attention maps to augment CNN features. This allows the CNN to capture rich local information while being enhanced by global context from the Transformer, leading to more discriminative features for polyp segmentation.

SBA-Net integrates these components by using multiple TAC modules in the encoder to generate boosted CNN features, which are then fed into SBA modules in the decoder for progressive feature enhancement and segmentation.

Datasets

The paper evaluates SBA-Net on several colonoscopy image datasets:

  • CVC-ClinicDB
  • Kvasir
  • CVC-ColonDB
  • ETIS
  • CVC-300

The training and testing splits for CVC-ClinicDB and Kvasir are aligned with those used in the PraNet paper for fair comparison. Data augmentation techniques such as random scaling and color exchange are employed.

Key Insights and Ideas

  • The Dominance of Middle Frequencies: The most striking insight is the empirical finding that middle-frequency sub-bands in the spectral domain become increasingly significant as features propagate through CNN layers for polyp segmentation. This challenges the common intuition that only high frequencies are important for details and low frequencies for context.
  • Sub-Band Attention for Discrimination: The paper proposes that focusing attention on these specific, informative sub-bands (middle and high) can significantly improve feature discrimination, leading to more robust segmentation.
  • Synergy of CNNs and Transformers: The research highlights the benefit of combining CNNs and Transformers, not just by fusing their outputs, but by using Transformer features as attention mechanisms to guide and enhance CNN feature extraction (as in the TAC module). This approach aims to retain the local detail richness of CNNs while incorporating the global context awareness of Transformers.

Key Results

  • State-of-the-Art Performance: SBA-Net consistently outperforms several state-of-the-art methods across multiple datasets and evaluation metrics (DSC, IoU, F-measure, S-measure, E-measure, MAE). For instance, on CVC-ClinicDB and Kvasir, SBA-Net is reported as the best in eight out of twelve metrics and second best in the remaining four.
  • Robustness to Challenging Cases: Qualitative results (Figure 8) demonstrate SBA-Net's effectiveness on images with very blurry polyp boundaries and low contrast, where other methods struggle.
  • Effectiveness of Components: Ablation studies (Table 3) show that both the SBA module and the TAC encoder contribute significantly to the overall performance. The full SBA-Net architecture, combining both, yields the best results.

Limitations and Future Work

  • Fixed Gaussian Filters: The current SBA module uses fixed Gaussian filters to extract sub-bands. The authors suggest that an adaptive approach to sub-band extraction could further enhance its applicability to a wider range of tasks.
  • Hadamard Product Fusion: While the TAC module effectively fuses CNN and Transformer features using a Hadamard product, the paper posits that more sophisticated fusion operations could potentially lead to even better performance.
  • Future Directions: The authors suggest exploring adaptive sub-band extraction and more advanced fusion mechanisms as avenues for future research to further improve polyp segmentation robustness.

Summary updated

BibTeX
@inproceedings{fang2023sub,
  author = {Xianyong Fang and Yuqing Shi and Qingqing Guo and Linbo Wang and Zhengyi Liu},
  title = {{Sub-Band Based Attention for Robust Polyp Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/82},
}
Figure