SBSS: Stacking-Based Semantic Segmentation Framework for Very High Resolution Remote Sensing Image

Yuanzhi Cai, Lei Fan, Yuan Fang

Abstract

Semantic segmentation of Very High Resolution (VHR) remote sensing images is a fundamental task for many applications. However, large variations in the scales of objects in those VHR images pose a challenge for performing accurate semantic segmentation. Existing semantic segmentation networks are able to analyse an input image at up to four resizing scales, but this may be insufficient given the diversity of object scales. Therefore, Multi Scale (MS) test-time data augmentation is often used in practice to obtain more accurate segmentation results, which makes equal use of the segmentation results obtained at the different resizing scales. However, it was found in this study that different classes of objects had their preferred resizing scale for more accurate semantic segmentation. Based on this behaviour, a Stacking-Based Semantic Segmentation (SBSS) framework is proposed to improve the segmentation results by learning this behaviour, which contains a learnable Error Correction Module (ECM) for segmentation result fusion and an Error Correction Scheme (ECS) for computational complexity control. Two ECS, i.e., ECS-MS and ECS-SS, are proposed and investigated in this study. The Floating-point operations (Flops) required for ECS-MS and ECS-SS are similar to the commonly used MS test and the Single-Scale (SS) test, respectively. Extensive experiments on four datasets (i.e., Cityscapes, UAVid, LoveDA and Potsdam) show that SBSS is an effective and flexible framework. It achieved higher accuracy than MS when using ECS-MS, and similar accuracy as SS with a quarter of the memory footprint when using ECS-SS.

Introduction

In Very High Resolution (VHR) remote sensing—captured by drones or satellites—a single image can contain objects at vastly different scales, from a tiny car to a massive industrial complex. Standard semantic segmentation networks have a fixed "receptive field," meaning they can only "see" a certain amount of context at once.

To handle this, researchers typically use Multi-Scale (MS) testing, where an image is resized several times, run through a network, and the results are averaged. However, this study identifies a critical flaw: averaging assumes all scales are equally good for all objects. In reality, large objects (like buildings) are often segmented better when the image is shrunk to fit the network's field of view, while small objects (like pedestrians) require zooming in to maintain detail. Simple averaging washes out these class-specific strengths.

The SBSS Framework

The proposed Stacking-Based Semantic Segmentation (SBSS) framework treats multi-scale fusion as a "stacking" ensemble problem—learning how to combine predictions rather than just averaging them. The framework operates through an iterative correction process that starts with a segmentation map at the smallest scale and refines it using information from progressively larger scales.

The overall SBSS framework illustrating the iterative correction process and the selection of areas for refinement.

Error Correction Schemes (ECS)

To manage computational complexity, the authors introduce two schemes:

  • ECS-MS: Designed for maximum accuracy, processing all image patches across several scales.
  • ECS-SS: Designed for memory-constrained environments. It only processes "uncertain" patches at higher resolutions, allowing it to use much smaller input patches (reducing GPU memory) without sacrificing accuracy.

Error Correction Module (ECM)

The core engine of the framework is the Error Correction Module (ECM). It consists of two primary components:

  1. Error Correction Network (ECN): A small convolutional sub-network (inspired by ConvNeXt) that takes the probability maps from two different scales and learns to produce a refined, "corrected" map.
  2. Adaptive Confidence Threshold (ACT): A mechanism that compares the confidence of the original prediction versus the corrected one, only updating pixels where the new prediction is significantly more reliable.
Detailed architecture of the Error Correction Module (ECM), featuring the ECN and the Adaptive Confidence Threshold mechanism.

Experimental Evaluation

The framework was validated on four diverse benchmarks: Cityscapes (urban street-level), UAVid (drone-based oblique), LoveDA (urban/rural land cover), and Potsdam (high-resolution aerial).

Key Insights

  • Class-Scale Bias: The research quantitatively proved that classes have "preferred" scales. For example, in Cityscapes, "Road" and "Sidewalk" performed best at 0.75x scale, while "Person" and "Bicycle" peaked at 1.5x or 1.75x.
  • Learnable Fusion vs. Averaging: By using the ECN to fuse scales, the model learns to trust small-scale predictions for context and large-scale predictions for fine details.
  • Efficiency: The ECS-SS results show that high-resolution passes are only necessary for areas where the low-resolution model is uncertain, saving significant GPU memory.

Results

  • Accuracy: On the Cityscapes test set, SBSS-MS achieved 82.6% mIoU, outperforming standard MS testing and state-of-the-art models like HRNet and SegFormer.
  • Memory Efficiency: SBSS-SS achieved comparable accuracy to standard single-scale (SS) testing but used 75% less GPU memory by utilizing smaller patches and selective correction.
  • Qualitative Improvement: Visual results show that SBSS significantly reduces fragmentation in large objects and produces cleaner boundaries compared to standard averaging.

Conclusion

SBSS offers a flexible and effective solution for multi-scale semantic segmentation in VHR imagery. While currently using predefined scales, future work may involve adaptive scale selection based on image content and the incorporation of edge-aware techniques to further refine object boundaries. Optimizing the backbone for multi-scale efficiency remains a key challenge for future research.

Summary updated

Abstract

Semantic segmentation of Very High Resolution (VHR) remote sensing images is a fundamental task for many applications. However, large variations in the scales of objects in those VHR images pose a challenge for performing accurate semantic segmentation. Existing semantic segmentation networks are able to analyse an input image at up to four resizing scales, but this may be insufficient given the diversity of object scales. Therefore, Multi Scale (MS) test-time data augmentation is often used in practice to obtain more accurate segmentation results, which makes equal use of the segmentation results obtained at the different resizing scales. However, it was found in this study that different classes of objects had their preferred resizing scale for more accurate semantic segmentation. Based on this behaviour, a Stacking-Based Semantic Segmentation (SBSS) framework is proposed to improve the segmentation results by learning this behaviour, which contains a learnable Error Correction Module (ECM) for segmentation result fusion and an Error Correction Scheme (ECS) for computational complexity control. Two ECS, i.e., ECS-MS and ECS-SS, are proposed and investigated in this study. The Floating-point operations (Flops) required for ECS-MS and ECS-SS are similar to the commonly used MS test and the Single-Scale (SS) test, respectively. Extensive experiments on four datasets (i.e., Cityscapes, UAVid, LoveDA and Potsdam) show that SBSS is an effective and flexible framework. It achieved higher accuracy than MS when using ECS-MS, and similar accuracy as SS with a quarter of the memory footprint when using ECS-SS.
BibTeX
@misc{cai2022sbss,
  author = {Yuanzhi Cai and Lei Fan and Yuan Fang},
  title = {{SBSS: Stacking-Based Semantic Segmentation Framework for Very High Resolution Remote Sensing Image}},
  year = {2022},
  eprint = {2212.07623},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.07623},
}
Figure