Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze & Excitation' Blocks

Abhijit Guha Roy, Nassir Navab, Christian Wachinger

Abstract

This paper introduces a novel architectural enhancement for medical image segmentation by integrating "Squeeze & Excitation" (SE) blocks into Fully Convolutional Neural Networks (F-CNNs). By adaptively recalibrating feature maps along both channel and spatial dimensions, the proposed Concurrent Spatial and Channel SE (scSE) block significantly improves segmentation accuracy. Evaluations on three diverse medical imaging datasets (MRI, CT, and OCT) demonstrate that adding scSE blocks to standard architectures like U-Net yields a 4-9% boost in Dice scores with a negligible 1.5% increase in model parameters, outperforming simple increases in network depth.

Introduction

In the domain of semantic segmentation, Fully Convolutional Neural Networks (F-CNNs) such as U-Net are the standard. However, standard convolutions treat all channels and spatial pixels with equal importance within a local neighborhood. They lack an inherent mechanism to "recalibrate" themselves to ignore noise or highlight specific anatomical structures that may be small or irregularly shaped. This research addresses whether networks can learn to identify which features are task-relevant and suppress the rest through adaptive recalibration.

Methodology

The authors build upon the original "Squeeze & Excitation" (SE) concept—initially designed for classification—and adapt it for the pixel-level requirements of segmentation. Three variants are proposed:

  1. Channel SE (cSE): This block "squeezes" spatial information into a single vector using global average pooling and then "excites" (reweights) the channels. This provides a global view of the image to determine which features are globally important (the "what").
  2. Spatial SE (sSE): A novel contribution that "squeezes" channel information using a $1 \times 1$ convolution to create a single-channel spatial map. This acts as an attention mechanism, identifying where the network should focus spatially (the "where").
  3. Concurrent Spatial and Channel SE (scSE): This combines both blocks. Among various merging strategies, Max-Out—taking the maximum value between the two blocks at every pixel—was found to be most effective, as it encourages competition between spatial and channel importance.
Architectural diagrams of (a) cSE, (b) sSE, and (c) scSE blocks.

These blocks are designed to be "plug-and-play," allowing integration after encoder and decoder blocks in standard architectures like U-Net, SD-Net, and FC-DenseNet.

Experimental Setup

The approach was validated on three challenging medical imaging datasets:

  • MALC (Brain MRI): Segmenting 27 cortical and sub-cortical structures, characterized by severe class imbalance.
  • Visceral (Whole-body CT): Segmenting 10 abdominal organs with highly variable shapes.
  • Duke (Retinal OCT): Segmenting 7 thin retinal layers (5-6 pixels wide) amidst significant noise.
Various placement strategies (P1-P6) for integrating SE blocks within an F-CNN architecture.

The researchers explored several placement strategies (P1-P6), ultimately finding that placing blocks throughout the encoder and decoder paths (P5/P6) yielded the best results.

Key Results and Discussion

  • Performance Gains: On the MALC dataset, the Dice score for U-Net improved from 0.763 to 0.851, representing an ~8.8% absolute improvement. This consistency was observed across all tested architectures and modalities.
  • Efficiency vs. Depth: The scSE block proved far more efficient than increasing network depth. For instance, adding an extra encoder/decoder block to SD-Net increased complexity by 33% for a 5% gain, whereas scSE provided a 9% gain for only 1.5% more parameters.
  • Small Structure Recovery: Qualitatively, SE blocks enabled the recovery of tiny structures (e.g., the amygdala) that baseline models often missed. It also reduced "over-segmentation" where the model would erroneously bleed into adjacent tissues.
  • Competitive Aggregation: The superiority of the Max-Out strategy suggests that letting spatial and channel excitations compete for dominance leads to more discriminative features than simple addition.

Conclusion

The integration of spatial and channel SE blocks provides a powerful, parameter-efficient method for boosting F-CNN performance in medical imaging. While highly effective, future work may explore the sensitivity of the bottleneck ratio ($r$) and the extension of these concurrent blocks to fully 3D volumetric data.

Summary updated

Abstract

In a wide range of semantic segmentation tasks, fully convolutional neural networks (F-CNNs) have been successfully leveraged to achieve state-of-the-art performance. Architectural innovations of F-CNNs have mainly been on improving spatial encoding or network connectivity to aid gradient flow. In this article, we aim towards an alternate direction of recalibrating the learned feature maps adaptively; boosting meaningful features while suppressing weak ones. The recalibration is achieved by simple computational blocks that can be easily integrated in F-CNNs architectures. We draw our inspiration from the recently proposed 'squeeze & excitation' (SE) modules for channel recalibration for image classification. Towards this end, we introduce three variants of SE modules for segmentation, (i) squeezing spatially and exciting channel-wise, (ii) squeezing channel-wise and exciting spatially and (iii) joint spatial and channel 'squeeze & excitation'. We effectively incorporate the proposed SE blocks in three state-of-the-art F-CNNs and demonstrate a consistent improvement of segmentation accuracy on three challenging benchmark datasets. Importantly, SE blocks only lead to a minimal increase in model complexity of about 1.5%, while the Dice score increases by 4-9% in the case of U-Net. Hence, we believe that SE blocks can be an integral part of future F-CNN architectures.
BibTeX
@article{roy2019recalibrating,
  author = {Abhijit Guha Roy and Nassir Navab and Christian Wachinger},
  title = {{Recalibrating Fully Convolutional Networks with Spatial and Channel 'Squeeze \& Excitation' Blocks}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2019},
  doi = {10.1109/tmi.2018.2867261},
}
Figure