Pyramid-Net: Intra-layer Pyramid-Scale Feature Aggregation Network for Retinal Vessel Segmentation

Jiawei Zhang, Yanchun Zhang, Hailong Qiu, Wen Xie, Zeyang Yao, Haiyun Yuan, Qianjun Jia, Tianchen Wang, Yiyu Shi, Meiping Huang, Jian Zhuang, Xiaowei XuFrontiers in Medicine 2021 · DOI

Abstract

Retinal vessel segmentation plays an important role in the diagnosis of eye-related diseases and biomarkers discovery. Existing works perform multi-scale feature aggregation in an inter-layer manner, which may result in limited performance, especially on thin vessels. We propose Pyramid-Net, featuring intra-layer pyramid-scale aggregation blocks (IPABs) to fuse multi-scale features within each layer. Enhanced by pyramid inputs, deep supervision, and pyramid skip connections, Pyramid-Net achieves state-of-the-art performance on DRIVE, STARE, and CHASE-DB1 datasets. Experimental results show significant improvements in thin vessel segmentation and a large reduction in computational cost.

Introduction

Retinal vessel segmentation is a critical task for diagnosing systemic diseases like diabetes, hypertension, and glaucoma. While deep learning has made massive strides, "thin vessels" (those only 1–2 pixels wide) remain a major hurdle. Standard Convolutional Neural Networks (CNNs) rely on pooling and downsampling to capture high-level context, but this process often destroys the delicate structural information of small vessels.

Retinal fundus image with regions of interest highlighted for thick (yellow) and thin (green) vessel analysis.

Existing "multi-scale" solutions usually perform inter-layer aggregation—fusing features from a low-resolution layer with a high-resolution one. The authors argue this is insufficient because it only considers features at the specific scales of those layers. Pyramid-Net shifts the focus to intra-layer aggregation, ensuring that every stage of the network processes the "big picture" and "fine details" simultaneously.

Methods

The core of the architecture is the Intra-layer Pyramid-scale Aggregation Block (IPAB). Built on the foundation of a ResNet block, the IPAB splits the feature map into three parallel branches:

  1. Current Scale: The standard processing path.
  2. Higher Scale: Uses up-sampling to capture finer details.
  3. Lower Scale: Uses down-sampling to capture broader context.

To maintain efficiency, channel counts are reduced in auxiliary branches. Three additional enhancements boost performance:

  • Pyramid Input Enhancement: Resized versions of the original image are fed directly into encoder IPABs to prevent initial information loss.
  • Deep Pyramid Supervision: Loss is calculated at multiple scales within the decoder, forcing the network to learn meaningful features at every resolution.
  • Pyramid Skip Connections: An evolution of standard U-Net skip connections allowing the decoder to reuse encoder features across multiple scales (from $l-2$ to $l+2$).

Experimental Results

Pyramid-Net was evaluated on three gold-standard datasets: DRIVE (40 images), STARE (20 images with pathological cases), and CHASE-DB1 (28 images with varied illumination).

The model achieved state-of-the-art (SOTA) performance across all benchmarks. On the DRIVE dataset, it reached an Accuracy of 96.26% and an AUC of 98.32%. Notably, thin vessel segmentation improved by 4.73% over the backbone CE-Net model.

Comparison of segmentation results for thick and thin vessels, demonstrating Pyramid-Net's ability to preserve fine structural details compared to ground truth.

Beyond accuracy, Pyramid-Net is highly efficient. By utilizing channel pruning in auxiliary branches, it achieves a 64.7% reduction in FLOPs compared to U-Net++, requiring only 188.15G FLOPs, making it much faster and lighter than competitors like CB-Net (441.62G).

Conclusion

Pyramid-Net addresses the limitations of inter-layer feature aggregation by introducing intra-layer multi-scale processing. While it excels at thin vessel detection, challenges remain for extremely low-contrast vessels in highly pathological images. Future work will focus on improving robustness to thickness bias and enhancing performance in complex clinical scenarios.

Summary updated

Abstract

Retinal vessel segmentation plays an important role in the diagnosis of eye-related diseases and biomarkers discovery. Existing works perform multi-scale feature aggregation in an inter-layer manner, namely <b>inter-layer feature aggregation</b>. However, such an approach only fuses features at either a lower scale or a higher scale, which may result in a limited segmentation performance, especially on thin vessels. This discovery motivates us to fuse multi-scale features in each layer, <b>intra-layer feature aggregation</b>, to mitigate the problem. Therefore, in this paper, we propose Pyramid-Net for accurate retinal vessel segmentation, which features intra-layer pyramid-scale aggregation blocks (IPABs). At each layer, IPABs generate two associated branches at a higher scale and a lower scale, respectively, and the two with the main branch at the current scale operate in a <b>pyramid-scale</b> manner. Three further enhancements including pyramid inputs enhancement, deep pyramid supervision, and pyramid skip connections are proposed to boost the performance. We have evaluated Pyramid-Net on three public retinal fundus photography datasets (DRIVE, STARE, and CHASE-DB1). The experimental results show that Pyramid-Net can effectively improve the segmentation performance especially on thin vessels, and outperforms the current state-of-the-art methods on all the adopted three datasets. In addition, our method is more efficient than existing methods with a large reduction in computational cost. We have released the source code at https://github.com/JerRuy/Pyramid-Net.
BibTeX
@inproceedings{zhang2021pyramid,
  author = {Jiawei Zhang and Yanchun Zhang and Hailong Qiu and Wen Xie and Zeyang Yao and Haiyun Yuan and Qianjun Jia and Tianchen Wang and Yiyu Shi and Meiping Huang and Jian Zhuang and Xiaowei Xu},
  title = {{Pyramid-Net: Intra-layer Pyramid-Scale Feature Aggregation Network for Retinal Vessel Segmentation}},
  booktitle = {Frontiers in Medicine},
  year = {2021},
  doi = {10.3389/fmed.2021.761050},
}
Figure