Knowledge and Geo-Object Based Graph Convolutional Network for Remote Sensing Semantic Segmentation

Wei Cui, Yao Meng, Yuanjie Hao, Ziwei Wang, Xin He, Weijie Wu, Jie Li, Huilin Zhao, Cong Xia, Jin WangSensors 2021 · DOI

TL;DR

This paper introduces the Knowledge and Geo-Object Based Graph Convolutional Network (KGGCN) for remote sensing semantic segmentation. KGGCN addresses limitations of traditional pixel-based and graph-based methods by integrating prior geographic knowledge with spatial correlations, leading to improved accuracy and robustness in segmenting complex scenes.

What Problem Does This Paper Address?

Semantic segmentation of remote sensing images is crucial for understanding our planet, but existing methods face significant challenges. Pixel-based approaches, while powerful, struggle to capture the relationships between geographic objects and often suffer from the "salt-and-pepper" effect (isolated misclassified pixels). Object-based methods, like those using Graph Neural Networks (GNNs), represent objects as nodes. However, standard GNNs rely heavily on local contextual information within a single image sample. This can be problematic because this local context can be distorted or incomplete, leading to misclassifications. For instance, objects with similar spectral features but different geographic meanings (e.g., city grass vs. flat field) or objects with incomplete surrounding information due to sample clipping can confuse these models. The paper aims to overcome these limitations by incorporating broader, "geo-object prior knowledge" from the entire study area.

Methods

The proposed KGGCN model is an object-based approach that leverages Graph Convolutional Networks (GCNs) enhanced with prior geographic knowledge. The core innovation lies in how it integrates this knowledge.

Here's a breakdown of the key components:

  • Superpixel Segmentation: The initial step divides the remote sensing image into superpixel blocks, which serve as the nodes in the graph. This inherently addresses the salt-and-pepper issue by grouping pixels into meaningful objects.
  • Node Feature Extraction: A CNN extracts initial features from the image, which are then associated with each superpixel node.
  • Spatial Correlation Recognition Module: This module learns the spatial relationships between neighboring superpixel nodes. It constructs a spatial adjacency matrix that quantifies how "close" or "related" nodes are based on their proximity.
  • KGGCN Module (Knowledge and Geo-Object Based Graph Convolutional Network): This is the heart of the proposed method. It takes the node features, the spatial adjacency matrix, and prior geographic knowledge as input.
  • Prior Knowledge Embedding: The prior knowledge is derived from the co-occurrence matrix, which captures the statistical probability of different land cover categories appearing together across the entire dataset. This matrix is embedded into the graph structure by creating a new, expanded graph where nodes are duplicated for each possible category. Edges in this new graph are then formed by combining spatial correlations with co-occurrence probabilities.
  • Feature Aggregation: Graph convolution is performed on this enriched graph. This allows nodes to aggregate information not just from their immediate neighbors within a single sample but also from the broader geographic context encoded in the co-occurrence matrix. This effectively extends the receptive field of each node.
  • Graph Restoration: After aggregation, the expanded graph is restored to the original graph structure, but with updated node features that now incorporate both spatial and prior knowledge.
  • Classifier: A final fully connected layer uses the updated node features to predict the semantic class for each superpixel.

Datasets

The paper uses a remote sensing dataset from Wenchuan County, Sichuan Province, China, for training and validation. This dataset consists of high-resolution remote sensing images and corresponding ground truth labels.

For supplementary experiments to evaluate generalization, the Indian Pines dataset was used for Hyperspectral Image (HSI) classification. This dataset is a standard benchmark in HSI research.

Key Insights and Ideas

  • Beyond Local Context: The most significant insight is that relying solely on local image context for GNNs in remote sensing is insufficient. Distorted or incomplete local information can lead to misclassifications.
  • The Power of Co-occurrence: The co-occurrence matrix, derived from the entire dataset, acts as a powerful form of "geo-object prior knowledge." It captures how different land cover types typically relate to each other across a region, providing a global perspective that local context lacks.
  • Bridging the Gap: KGGCN effectively bridges the gap between object-level understanding and global geographic relationships by embedding co-occurrence probabilities into the graph structure. This allows the model to learn more robust and contextually aware representations.
  • Solving "Reversal of Geography": The method directly tackles issues like "different objects with the same spectrum" and "scene distortion" by down-weighting irrelevant neighbors and emphasizing meaningful spatial and categorical relationships.

Key Results

  • Improved Pixel Accuracy: KGGCN achieved a pixel accuracy of 0.9074, which is 4.1% higher than U-Net and 3.7% higher than the baseline Cluster-GCN.
  • Superior Object Classification: For object classification accuracy, KGGCN reached 0.8496, outperforming Cluster-GCN by 8.3%.
  • Better Performance on Difficult Classes: The model showed significant improvements in classifying challenging classes like "city_grass" (accuracy of 0.764) and "city_forest" (accuracy of 0.833), where other models struggled.
  • Efficiency: KGGCN also demonstrated advantages in model size, memory consumption, and inference speed compared to U-Net, while being comparable to Cluster-GCN.
  • HSI Classification: While KGGCN showed good performance on the Indian Pines HSI dataset, it was slightly inferior to state-of-the-art methods, which the authors attribute to the different nature of HSI data (spectral vs. spatial dominance) and the difficulty in obtaining reliable co-occurrence probabilities from a single image.

Limitations and Future Work

  • HSI Generalization: The paper acknowledges that KGGCN's performance on Hyperspectral Images (HSI) was not as strong as on high-resolution remote sensing images. This is attributed to the spectral nature of HSI data and the challenge of deriving meaningful co-occurrence probabilities from a single image.
  • Scale of Segmentation: The authors note that different geo-objects have varying scales, and future work should focus on balancing segmentation scales.
  • Automatic Knowledge Acquisition: The current method relies on manual statistics for prior knowledge. Future research will explore automatic learning and adjustment methods for acquiring this knowledge more efficiently and objectively.
  • 3D Convolution for HSI: For HSI, the authors suggest exploring 3D convolution to better extract spectral features and integrating prior geographic knowledge with HSI data.

Summary updated

Abstract

Pixel-based semantic segmentation models fail to effectively express geographic objects and their topological relationships. Therefore, in semantic segmentation of remote sensing images, these models fail to avoid salt-and-pepper effects and cannot achieve high accuracy either. To solve these problems, object-based models such as graph neural networks (GNNs) are considered. However, traditional GNNs directly use similarity or spatial correlations between nodes to aggregate nodes' information, which rely too much on the contextual information of the sample. The contextual information of the sample is often distorted, which results in a reduction in the node classification accuracy. To solve this problem, a knowledge and geo-object-based graph convolutional network (KGGCN) is proposed. The KGGCN uses superpixel blocks as nodes of the graph network and combines prior knowledge with spatial correlations during information aggregation. By incorporating the prior knowledge obtained from all samples of the study area, the receptive field of the node is extended from its sample context to the study area. Thus, the distortion of the sample context is overcome effectively. Experiments demonstrate that our model is improved by 3.7% compared with the baseline model named Cluster GCN and 4.1% compared with U-Net.
BibTeX
@inproceedings{cui2021knowledge,
  author = {Wei Cui and Yao Meng and Yuanjie Hao and Ziwei Wang and Xin He and Weijie Wu and Jie Li and Huilin Zhao and Cong Xia and Jin Wang},
  title = {{Knowledge and Geo-Object Based Graph Convolutional Network for Remote Sensing Semantic Segmentation}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21113848},
}
Figure