Abstract
Semantic information can ensure better pixel classification, and the spatial information of the low-level feature map can ensure the detailed location of the pixels. However, this part of spatial information is often ignored in capturing semantic information, which is a significant loss for the spatial location of the image semantic category itself. To better alleviate this problem, we propose a Long and Short-Range Relevance Context Network (LSRRCNet). Specifically, we first construct a Long-Range Relevance Context Module (LRRC) to capture the global semantic context of the high-level features and the ignored local spatial context information. Simultaneously, we build a Short-Range Relevance Context Module (SRRC) to capture the piecewise spatial context information in each stage of the low-level features in the form of jump connections. The whole network adopts a coding and decoding structure to better improve the segmentation results. Finally, we conduct a large number of experiments on three semantic segmentation datasets (PASCAL VOC2012, Cityscapes, and ADE20K) to verify the effectiveness of the network.
Introduction
Semantic segmentation requires a delicate balance: semantic information is necessary to identify what an object is (classification), while spatial information is required to determine exactly where its boundaries are (localization).
Traditional Convolutional Neural Networks (CNNs) struggle with this balance. As an image passes through deeper layers, spatial resolution is often sacrificed for high-level semantics. This typically leads to "fuzzy" boundaries or the complete disappearance of small objects. Conversely, shallow layers possess great spatial detail but lack the "big picture" context needed to distinguish between similar-looking categories. This research identifies the lack of "relevance context"—the relationship between a pixel and its neighbors (short-range) and the rest of the image (long-range)—as the primary bottleneck in current models.
Methodology
The proposed LSRRCNet is an encoder-decoder framework using ResNet as a backbone, featuring two key innovations:
-
Long-Range Relevance Context (LRRC) Module: Applied to the final high-level feature maps, this module uses a dual-path approach. One path uses strip pooling (horizontal and vertical) combined with a self-attention mechanism to capture global dependencies without the noise often introduced by standard global pooling. The second path uses dilated convolutions to capture long-range local context.
-
Short-Range Relevance Context (SRRC) Module: Integrated into the earlier, low-level stages of the encoder, the SRRC captures "piecewise" spatial information. It uses a mix of standard and dilated convolutions to increase the receptive field slightly while maintaining high resolution. These refined spatial features are then passed to the decoder via jump connections to help reconstruct sharp object boundaries.
Experimental Setup
The model was evaluated on three gold-standard benchmarks:
- PASCAL VOC2012: A classic dataset with 20 foreground object classes.
- Cityscapes: A high-resolution dataset focused on urban street scenes across 50 cities.
- ADE20K: A challenging dataset containing 150 fine-grained semantic concepts across diverse scenes.
Key Insights
- Strip Pooling vs. Global Pooling: Standard global average pooling can be too "blunt," mixing features from unrelated regions. Strip pooling allows the model to capture long-range dependencies along specific axes, which is particularly useful for elongated objects like roads or poles.
- Refining Jump Connections: In many architectures (like U-Net), jump connections pass raw low-level features directly to the decoder. LSRRCNet argues that these features should be processed through the SRRC module first to extract relevant spatial context, leading to much cleaner segmentation edges.
Results and Discussion
LSRRCNet consistently outperformed several high-profile models, including DeepLabv3+, PSPNet, and DANet. It achieved 78.18% mIoU on PASCAL VOC2012 and 78.65% mIoU on Cityscapes.
Qualitative visualizations demonstrate that LSRRCNet is significantly better at preserving thin structures (such as horse legs) and distinguishing between overlapping objects (such as pedestrians in front of buses) compared to traditional Fully Convolutional Networks (FCNs).
Conclusion
While LSRRCNet performs excellently on standard benchmarks, complex background segmentation remains a challenge, particularly in scenes with extreme occlusions or highly irregular shapes. Future research will focus on developing more sophisticated pixel-to-pixel relationship models and optimizing the network for real-time performance without sacrificing accuracy.
Summary updated