Long and short-range relevance context network for semantic segmentation

Qing Liu, Yongsheng Dong, Yuanhua Pei, Lintao Zheng, Lei ZhangComplex 2023 · DOI

Abstract

Semantic information can ensure better pixel classification, and the spatial information of the low-level feature map can ensure the detailed location of the pixels. However, this part of spatial information is often ignored in capturing semantic information, which is a significant loss for the spatial location of the image semantic category itself. To better alleviate this problem, we propose a Long and Short-Range Relevance Context Network (LSRRCNet). Specifically, we first construct a Long-Range Relevance Context Module (LRRC) to capture the global semantic context of the high-level features and the ignored local spatial context information. Simultaneously, we build a Short-Range Relevance Context Module (SRRC) to capture the piecewise spatial context information in each stage of the low-level features in the form of jump connections. The whole network adopts a coding and decoding structure to better improve the segmentation results. Finally, we conduct a large number of experiments on three semantic segmentation datasets (PASCAL VOC2012, Cityscapes, and ADE20K) to verify the effectiveness of the network.

Introduction

Semantic segmentation requires a delicate balance: semantic information is necessary to identify what an object is (classification), while spatial information is required to determine exactly where its boundaries are (localization).

Traditional Convolutional Neural Networks (CNNs) struggle with this balance. As an image passes through deeper layers, spatial resolution is often sacrificed for high-level semantics. This typically leads to "fuzzy" boundaries or the complete disappearance of small objects. Conversely, shallow layers possess great spatial detail but lack the "big picture" context needed to distinguish between similar-looking categories. This research identifies the lack of "relevance context"—the relationship between a pixel and its neighbors (short-range) and the rest of the image (long-range)—as the primary bottleneck in current models.

Methodology

The proposed LSRRCNet is an encoder-decoder framework using ResNet as a backbone, featuring two key innovations:

  1. Long-Range Relevance Context (LRRC) Module: Applied to the final high-level feature maps, this module uses a dual-path approach. One path uses strip pooling (horizontal and vertical) combined with a self-attention mechanism to capture global dependencies without the noise often introduced by standard global pooling. The second path uses dilated convolutions to capture long-range local context.

  2. Short-Range Relevance Context (SRRC) Module: Integrated into the earlier, low-level stages of the encoder, the SRRC captures "piecewise" spatial information. It uses a mix of standard and dilated convolutions to increase the receptive field slightly while maintaining high resolution. These refined spatial features are then passed to the decoder via jump connections to help reconstruct sharp object boundaries.

Experimental Setup

The model was evaluated on three gold-standard benchmarks:

  • PASCAL VOC2012: A classic dataset with 20 foreground object classes.
  • Cityscapes: A high-resolution dataset focused on urban street scenes across 50 cities.
  • ADE20K: A challenging dataset containing 150 fine-grained semantic concepts across diverse scenes.

Key Insights

  • Strip Pooling vs. Global Pooling: Standard global average pooling can be too "blunt," mixing features from unrelated regions. Strip pooling allows the model to capture long-range dependencies along specific axes, which is particularly useful for elongated objects like roads or poles.
  • Refining Jump Connections: In many architectures (like U-Net), jump connections pass raw low-level features directly to the decoder. LSRRCNet argues that these features should be processed through the SRRC module first to extract relevant spatial context, leading to much cleaner segmentation edges.

Results and Discussion

LSRRCNet consistently outperformed several high-profile models, including DeepLabv3+, PSPNet, and DANet. It achieved 78.18% mIoU on PASCAL VOC2012 and 78.65% mIoU on Cityscapes.

Qualitative visualizations demonstrate that LSRRCNet is significantly better at preserving thin structures (such as horse legs) and distinguishing between overlapping objects (such as pedestrians in front of buses) compared to traditional Fully Convolutional Networks (FCNs).

Qualitative comparison of segmentation results between the Image, FCN baseline, and Ground Truth, highlighting the model's ability to capture fine details and overlapping objects.

Conclusion

While LSRRCNet performs excellently on standard benchmarks, complex background segmentation remains a challenge, particularly in scenes with extreme occlusions or highly irregular shapes. Future research will focus on developing more sophisticated pixel-to-pixel relationship models and optimizing the network for real-time performance without sacrificing accuracy.

Summary updated

Abstract

Abstract The semantic information can ensure better pixel classification, and the spatial information of the low-level feature map can ensure the detailed location of the pixels. However, this part of spatial information is often ignored in capturing semantic information, it is a huge loss for the spatial location of the image semantic category itself. To better alleviate this problem, we propose a Long and Short-Range Relevance Context Network. Specifically, we first construct a Long-Range Relevance Context Module to capture the global semantic context of the high-level feature and the ignored local spatial context information. At the same time, we build a Short-Range Relevance Context Module to capture the piecewise spatial context information in each stage of the low-level features in the form of jump connections. The whole network adopts a coding and decoding structure to better improve the segmentation results. Finally, we conduct a large number of experiments on three semantic segmentation datasets (PASCAL VOC2012, Cityscapes and ADE20K datasets) to verify the effectiveness of the network.
BibTeX
@inproceedings{liu2023long,
  author = {Qing Liu and Yongsheng Dong and Yuanhua Pei and Lintao Zheng and Lei Zhang},
  title = {{Long and short-range relevance context network for semantic segmentation}},
  booktitle = {Complex},
  year = {2023},
  doi = {10.1007/s40747-023-01103-6},
}
Figure