The Importance of Skip Connections in Biomedical Image Segmentation

Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, Chris Pal

Abstract

This paper investigates the synergy between "long" skip connections, which recover spatial information, and "short" skip connections, which facilitate gradient flow, to enable the training of extremely deep Fully Convolutional Networks (FCNs). By hybridizing the architectures of U-Net and ResNet, the authors demonstrate that deep models can achieve near state-of-the-art results in biomedical image segmentation using significantly fewer parameters and no complex post-processing.

Introduction

Biomedical image segmentation requires high-precision pixel-level labeling. Standard Convolutional Neural Networks (CNNs) typically utilize a "contracting" path that downsamples images to capture context; however, this process often discards fine-grained spatial information necessary for accurate boundary detection.

While Fully Convolutional Networks (FCNs) like the U-Net introduced "long" skip connections to recover this lost data, they still face the vanishing gradient problem. As networks grow deeper (reaching hundreds of layers), the training signal weakens as it propagates backward, making deep FCNs difficult to optimize. This research explores whether adding "short" skip connections—residual connections—can unlock the potential of very deep architectures for medical imaging.

Methodology

The proposed architecture integrates two distinct types of skip connections to balance spatial recovery with training stability:

  1. Long Skip Connections: These link layers from the contracting (encoder) path directly to the expanding (decoder) path at corresponding resolutions, helping the model retain spatial details lost during downsampling.
  2. Short Skip Connections: Internal to the residual blocks, these allow gradients to bypass non-linearities, creating a "highway" for the gradient to flow uninterrupted to the earliest layers.
Figure 1: The hybrid FCN architecture (a) showing long skip connections, alongside various residual block designs (b, c, d) utilizing short skip connections.

The architecture employs three block types: bottleneck, basic, and simple. To maintain efficiency, $1 \times 1$ convolutions adjust filter counts, while simple decimation/repetition handles spatial adjustments. The study also compares standard Binary Cross-Entropy (BCE) loss with Dice Loss, the latter being more robust to the class imbalances typical in medical datasets.

Experimental Results

The model was evaluated on the EM ISBI 2012 dataset, consisting of electron microscopy images of the Drosophila larva ventral nerve cord. Despite having only 30 training images, the authors used extensive data augmentation (flipping, shearing, rotations, and spline warping) to improve generalization.

Figure 2: Segmentation results on the EM ISBI 2012 dataset. (a) Raw image, (b-d) predictions showing high boundary continuity.

The hybrid architecture achieved competitive scores on the EM challenge metrics:

  • Maximal foreground-restricted Rand score ($V_{rand}$): 0.969
  • Information theoretic score ($V_{info}$): 0.986

Notably, these results were achieved with only 11 million parameters—a third of the U-Net's 33 million—and without any post-processing (such as watershed transforms or conditional random fields) often required by other top-tier entries.

Key Insights and Discussion

  • The "Middle" Problem: Empirical evidence suggested that while long skip connections assist the shallowest and deepest parts of the network, they do not alleviate gradient issues in the middle layers. Short skip connections are essential for ensuring these central layers learn effectively.
  • Convergence and Stability: Models utilizing both skip connection types converged significantly faster and exhibited more stable updates during training.
  • Stochastic Inference: The use of test-time dropout (averaging multiple passes with dropout enabled) was found to further improve boundary continuity and prediction "cleanliness."

Conclusion

By combining long and short skip connections, the authors proved that architectural depth and clever gradient management can outperform raw model width. While the model does not yet surpass the absolute top of the leaderboard—which often relies on heavy ensembles and post-processing—it provides a highly efficient and powerful baseline for deep biomedical image segmentation. Future work will focus on extending these 2D architectures to 3D medical datasets.

Summary updated

Abstract

In this paper, we study the influence of both long and short skip connections on Fully Convolutional Networks (FCN) for biomedical image segmentation. In standard FCNs, only long skip connections are used to skip features from the contracting path to the expanding path in order to recover spatial information lost during downsampling. We extend FCNs by adding short skip connections, that are similar to the ones introduced in residual networks, in order to build very deep FCNs (of hundreds of layers). A review of the gradient flow confirms that for a very deep FCN it is beneficial to have both long and short skip connections. Finally, we show that a very deep FCN can achieve near-to-state-of-the-art results on the EM dataset without any further post-processing.
BibTeX
@inproceedings{drozdzal2016importance,
  author = {Michal Drozdzal and Eugene Vorontsov and Gabriel Chartrand and Samuel Kadoury and Chris Pal},
  title = {{The Importance of Skip Connections in Biomedical Image Segmentation}},
  booktitle = {LABELS/DLMIA @ MICCAI},
  year = {2016},
  doi = {10.1007/978-3-319-46976-8\_19},
}
Figure