Multifrequency PolSAR Image Fusion Classification Based on Semantic Interactive Information and Topological Structure

Yice Cao, Yan Wu, Ming Li, Mingjie Zheng, Peng Zhang, Jili WangIEEE TGRS 2023 · DOI

TL;DR

This paper introduces MF-STFnet, a novel deep learning framework for classifying land cover using multi-frequency Polarimetric Synthetic Aperture Radar (MF-PolSAR) imagery. It addresses the limitations of existing methods by effectively fusing complementary information across different frequency bands and integrating both local semantic features and nonlocal topological structures for improved accuracy and robustness.

What Problem Does This Paper Address?

Classifying land cover from radar imagery is crucial for Earth observation. While single-frequency PolSAR classification has seen significant advancements, multi-frequency PolSAR (MF-PolSAR) classification remains less explored. Existing deep learning approaches, primarily based on Convolutional Neural Networks (CNNs), tend to focus only on local spatial information, neglecting the broader, nonlocal relationships between different land cover types. This oversight limits their ability to fully leverage the rich, complementary information available across multiple radar frequencies, leading to suboptimal classification performance. The paper aims to bridge this gap by developing a method that effectively combines information from different frequency bands and captures both local and global contextual information.

Methods

The proposed MF-STFnet employs a multi-pronged approach:

  • Band-Specific Semantic Feature Extraction (BSFE): For each frequency band, a dedicated CNN extracts local semantic features. This preserves the unique characteristics of each band while ensuring diversity.
  • Cross-Band Interactive Feature Extraction Module (CIFEM): This novel module explicitly models the semantic correlations between different frequency bands. By calculating cross-band feature interactions, it extracts more discriminative features that capture the complementarity of the bands, enhancing the ability to differentiate between similar land cover types.
  • Semantic Information-Based Classification (SIC): The concatenated band-specific and cross-band interactive features are fed into a classification head (GAP, FC, Softmax) to produce semantic-based predictions. This leverages local spatial information and inter-band semantic relationships.
  • Topological Property-Based Classification (TPC): To capture nonlocal spatial information, the Graph Sample and Aggregate network (GraphSAGE) is utilized. It dynamically constructs subgraphs based on semantic features and aggregates neighborhood information to learn topological relationships between samples, improving robustness.
  • Adaptive Weighting Fusion (AWF): A novel strategy that adaptively fuses the classification outputs from SIC and TPC across all frequency bands. The weights are dynamically updated during training based on the total loss, allowing the model to learn the optimal contribution of each band and classification type.

Datasets

The paper evaluates MF-STFnet on three real-world MF-PolSAR datasets:

  • San Francisco: Dual-band (C-band and L-band) data with 5 land cover categories.
  • Woniupan: Dual-band (S-band and L-band) data with 5 land cover categories.
  • Flevoland: Three-band (C-, L-, and P-band) data with 15 land cover categories.

The authors use a chessboard-like sampling strategy for non-overlapping data partitioning to mitigate spatial correlation between training and testing sets.

Key Insights and Ideas

  • Complementarity is Key: The core idea is that different radar frequencies capture distinct aspects of the Earth's surface. Effectively fusing this complementary information is crucial for accurate land cover classification.
  • Beyond Local Features: Relying solely on local CNN features is insufficient. Incorporating nonlocal topological information, as captured by GraphSAGE, significantly enhances classification robustness.
  • Inter-Band Semantic Interaction: The CIFEM module is a critical innovation, explicitly modeling how features from different bands relate to each other semantically, leading to more discriminative representations.
  • Adaptive Fusion Matters: A static fusion strategy might not be optimal. The AWF strategy, which learns band weights adaptively based on the overall loss, allows for a more flexible and effective combination of information.
  • Joint Optimization: Training SIC and TPC collaboratively, with a consistency loss to encourage agreement, leads to a more robust and accurate overall model.

Key Results

  • Superior Performance: MF-STFnet consistently outperforms state-of-the-art methods across all three datasets, achieving higher Overall Accuracy (OA), Average Accuracy (AA), and Kappa coefficients.
  • On the San Francisco dataset, MF-STFnet achieved OA of 98.76%, AA of 98.69%, and Kappa of 0.9830.
  • On the Woniupan dataset, OA reached 95.41%, AA 93.02%, and Kappa 0.9290.
  • On the Flevoland dataset, OA was 95.45%, AA 92.36%, and Kappa 0.9477.
  • Ablation Studies: Ablation experiments confirmed the effectiveness of CIFEM and TPC. MF-STFnet, incorporating both modules, significantly outperformed models with only one or neither of these components.
  • Visual Improvements: t-SNE visualizations showed that MF-STFnet extracts more separable features compared to baseline methods. Classification maps also demonstrated better discrimination of challenging categories and reduced misclassifications.
  • Efficient Prediction: For prediction, the paper proposes using only the SIC results, which are computationally less demanding than TPC, to achieve efficient MF-PolSAR image classification.

Limitations and Future Work

  • Computational Cost: While MF-STFnet achieves superior performance, the inclusion of multiple deep learning modules (CNNs, CIFEM, GraphSAGE) can lead to higher computational demands compared to simpler methods.
  • Prediction Efficiency: The paper notes that TPC is computationally more expensive than SIC. While AWF is used for training, for efficient prediction, only SIC results are utilized. This suggests a potential trade-off between prediction speed and utilizing the full power of TPC during inference.
  • Future Work: The paper implicitly suggests that further exploration of adaptive fusion strategies and more efficient graph-based methods could lead to even better performance and reduced computational complexity. The adaptive weighting of AWF is highlighted as a key component for robust fusion, hinting at potential for further refinement in this area.

Summary updated

Abstract

Compared with the rapid development of single-frequency polarimetric SAR (PolSAR) image classification technology, there is less research on the land cover classification of multi-frequency PolSAR (MF-PolSAR) images. And the deep learning methods among them are mainly based on convolutional neural networks (CNNs), only local spatiality is considered but the nonlocal relationship is ignored. Therefore, this paper proposes the MF semantics and topology fusion (MF-STF) model based on semantic interaction and nonlocal topological structure to improve MF-PolSAR classification performance. During MF-STF optimization, the semantic information-based classification (SIC) and topological property-based classification (TPC) work collaboratively, not only fully leveraging the complementarity of bands, but also combining local and nonlocal spatial information to improve the discrimination of different categories. For SIC, the designed cross-band interactive feature extraction (CIFE) module is embedded to explicitly model the deep semantic correlation among bands, thereby leveraging the complementarity of bands to make ground objects more separable. In TPC, the graph sample and aggregate network (GraphSAGE) is employed to dynamically capture the representation of nonlocal topological relations between land cover categories. In this way, the robustness of classification can be further improved by combining nonlocal spatial information. Finally, a MF weighted fusion (MFWF) strategy is proposed to merge inference from different bands, so as to make the MF joint classification decisions of SIC and TPC. Notably, its weights are adjusted based on the total model loss. The effectiveness of the proposed modules is proved by ablation experiments on three measured MF-PolSAR datasets. In addition, the comparative experiments show that MF-STF can achieve more competitive classification performance than some state-of-the-art methods.
BibTeX
@article{cao2023multifrequency,
  author = {Yice Cao and Yan Wu and Ming Li and Mingjie Zheng and Peng Zhang and Jili Wang},
  title = {{Multifrequency PolSAR Image Fusion Classification Based on Semantic Interactive Information and Topological Structure}},
  journal = {IEEE Transactions on Geoscience and Remote Sensing},
  year = {2023},
  doi = {10.1109/tgrs.2023.3264560},
}
Figure