Prototype-Driven Structure Synergy Network for Remote Sensing Images Segmentation

Junyi Wang, Jinjiang Li, Guodong Fan, Yakun Ju, Xiang Fang, Alex C. Kot

TL;DR

This paper introduces PDSSNet, a novel network for remote sensing image segmentation that tackles the challenges of high intra-class variance and inter-class similarity. By synergizing semantic and structural information through adaptive prototypes and a hierarchical cognitive strategy, PDSSNet achieves more complete and accurate segmentation of ground objects.

What Problem Does This Paper Address?

Semantic segmentation of remote sensing images is crucial for applications like land cover mapping. However, accurately segmenting objects is difficult due to two main issues: high intra-class variance (objects of the same class looking very different due to scale, material, or lighting) and high inter-class similarity (objects of different classes looking very similar). Traditional methods often struggle to create unified representations for diverse objects within a class and to distinguish between similar-looking classes, leading to incomplete or incorrect segmentation. Existing class-guided methods, while improving, often rely on coarse prototypes or overlook the importance of structural information, further limiting their effectiveness.

Methods

PDSSNet is built upon a ConvNeXt backbone and introduces three key modules:

  • Adaptive Prototype Extraction Module (APEM): Instead of relying on model-generated coarse prototypes, APEM uses ground truth masks to directly aggregate features for each class. This creates more accurate and unbiased class prototypes, effectively addressing high intra-class variance.
  • Semantic-Structure Coordination Module (SSCM): This module adopts a hierarchical "semantics-first, structure-second" approach. It first establishes global semantic understanding and then refines it using structural information. This synergy ensures that semantic guidance aligns with the object's spatial structure, leading to more complete segmentation and preserving morphological integrity.
  • Channel Similarity Adjustment Module (CSAM): To tackle high inter-class similarity, CSAM employs a novel "SimStep" mechanism. This mechanism dynamically adjusts the update step size when encountering similar features, allowing the model to "skip" or compress short-term similar features and focus on discriminative characteristics.

Datasets

The paper evaluates PDSSNet on three standard remote sensing semantic segmentation benchmarks:

  • Vaihingen Dataset: Features high-resolution orthophotos with five land cover categories and a background class.
  • Potsdam Dataset: Offers high-resolution tiles with multispectral bands and a Digital Surface Model, also including several land cover categories.
  • LoveDA Dataset: Constructed from Google Earth imagery, spanning three Chinese cities and including seven land cover categories, known for its multi-scale objects, complex backgrounds, and imbalanced classes.

Key Insights and Ideas

  • Ground Truth is King for Prototypes: Directly using ground truth to generate class prototypes (APEM) is significantly more effective than relying on model-generated ones, especially for handling diverse object appearances within a class.
  • Structure is Key to Completeness: Simply focusing on semantic similarity isn't enough. Integrating structural information (SSCM) is crucial for ensuring that segmentation results are complete and maintain the correct object morphology.
  • Hierarchical Cognition for Segmentation: Mimicking human cognitive processes—understanding the general concept first, then refining with details—proves beneficial for complex image segmentation.
  • Dynamic Skipping for Similar Classes: A mechanism that allows the model to intelligently "skip" over highly similar features (CSAM) is vital for distinguishing between confusable classes.

Key Results

PDSSNet consistently outperforms state-of-the-art methods across all three datasets.

  • On the Vaihingen dataset, PDSSNet achieved a mean F1 score of 91.60% and mIoU of 84.68%.
  • On the Potsdam dataset, it reached a mean F1 score of 93.25% and mIoU of 87.55%.
  • On the LoveDA dataset, PDSSNet achieved a mean F1 score of 71.20% and mIoU of 56.10%.

Ablation studies demonstrated that removing any of the core modules (APEM, SSCM, CSAM) significantly degraded performance, highlighting their individual importance and synergistic effect. Visualizations showed PDSSNet producing more complete segmentations with clearer boundaries and fewer misclassifications, especially in challenging scenarios with high intra-class variance and inter-class similarity.

Limitations and Future Work

  • Backbone Choice: While PDSSNet performs well with various backbones, the paper notes that ConvNeXt-Tiny offers a good balance of performance and parameter count. Future work could explore other advanced backbones.
  • Computational Cost: While SSMs offer linear complexity, the overall network architecture might still be computationally intensive for very large-scale applications.
  • Generalizability: While tested on diverse datasets, further validation on a wider range of remote sensing scenarios (e.g., different sensor types, resolutions, and geographic regions) would be beneficial.

The authors express hope that PDSSNet will provide valuable insights for future research in incomplete class segmentation for remote sensing.

Summary updated

Abstract

In the semantic segmentation of remote sensing images, acquiring complete ground objects is critical for achieving precise analysis. However, this task is severely hindered by two major challenges: high intra-class variance and high inter-class similarity. Traditional methods often yield incomplete segmentation results due to their inability to effectively unify class representations and distinguish between similar features. Even emerging class-guided approaches are limited by coarse class prototype representations and a neglect of target structural information. Therefore, this article proposes a prototype-driven structure synergy network (PDSSNet). The design of this network is based on a core concept: a complete ground object is jointly defined by its invariant class semantics and its variant spatial structure. To implement this, we have designed three key modules. First, the adaptive prototype extraction module (APEM) ensures semantic accuracy from the source by encoding the ground truth to extract comprehensive class prototypes. Subsequently, the designed semantic-structure coordination module (SSCM) follows a hierarchical “semantics-first, structure-second” principle. This involves first establishing a global semantic cognition, then leveraging structural information to constrain and refine the semantic representation, thereby ensuring the integrity of class information. Finally, the channel similarity adjustment module (CSAM) employs a dynamic step-size adjustment mechanism to focus on discriminative features between classes. Extensive experiments demonstrate that PDSSNet outperforms state-of-the-art methods. The source code is available at https://github.com/wangjunyi-1/PDSSNet
BibTeX
@article{wang2025prototype,
  author = {Junyi Wang and Jinjiang Li and Guodong Fan and Yakun Ju and Xiang Fang and Alex C. Kot},
  title = {{Prototype-Driven Structure Synergy Network for Remote Sensing Images Segmentation}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2025},
  url = {https://openreview.net/forum?id=3scPGgH53o},
}

Loading related content...

Figure