SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification

Junyan Lin, Feng Gao, Xiaocheng Shi, Junyu Dong, Qian Du

Abstract

Multi-source remote sensing data fusion, such as combining Hyperspectral Imaging (HSI) with LiDAR or SAR, is essential for high-precision land-cover classification. This paper introduces SS-MAE, a self-supervised framework utilizing a dual-branch Masked Auto-Encoder to learn joint representations. By masking both spatial patches and spectral channels, the model captures complex global dependencies alongside local structural details. SS-MAE achieves state-of-the-art performance on several benchmarks, demonstrating the power of spatial-spectral pre-training.

Introduction

In remote sensing, the integration of diverse data sources—like the high spectral resolution of Hyperspectral Imaging (HSI) and the structural information from LiDAR or Synthetic Aperture Radar (SAR)—is key to accurate land-cover mapping. However, effectively fusing these modalities remains a significant challenge for deep learning models.

While Masked Image Modeling (MIM) has seen success in general computer vision, standard approaches often fail in remote sensing for two primary reasons: they neglect the high-dimensional spectral correlations inherent in HSI, and they struggle to balance global context with the fine-grained local details necessary for distinguishing similar land-cover classes. The Spatial-Spectral Masked Auto-Encoder (SS-MAE) addresses these issues through a novel self-supervised pre-training strategy.

The SS-MAE Framework

The core innovation of SS-MAE lies in its dual-branch masking strategy, designed to force the model to learn both spatial geometry and spectral signatures.

Dual-Branch Masking Strategy

Unlike traditional MAEs that only mask spatial patches, SS-MAE employs two parallel masking operations:

  1. Spatial-wise Masking: Randomly masks patches across all bands to teach the model about geometric structures and spatial context.
  2. Spectral-wise Masking: Masks entire spectral channels, forcing the model to reconstruct missing bands based on inter-band correlations. This is crucial for capturing the "chemical signature" of different materials.
Spatial-wise vs. Spectral-wise Masking Units in SS-MAE.

Architecture and Training

The SS-MAE architecture is a hybrid design that combines the strengths of Transformers and Convolutional Neural Networks (CNNs).

Hybrid Transformer-CNN Architecture

The model utilizes a dual-branch Transformer backbone. During the self-supervised pre-training stage, the encoders process only the visible (unmasked) tokens to reconstruct the original data. To overcome the Transformer's tendency to lose high-frequency local details (like edges), the authors integrated lightweight CNNs—specifically Inverted Residual Blocks—into the training and test stages. These CNNs run in parallel with the pre-trained encoders to capture fine-grained features.

Overall architecture of the SS-MAE framework across pre-training and testing stages.

Self-Supervised Pre-training

The model is first pre-trained on large-scale unlabeled multi-source data. By solving the "pre-text task" of reconstructing masked pixels and channels, the model develops a deep, generalized understanding of the Earth's surface before being fine-tuned on specific labeled datasets.

Experimental Evaluation

The researchers evaluated SS-MAE on three benchmark datasets: Berlin (HSI + SAR), Augsburg (HSI + SAR), and Houston 2018 (HSI + LiDAR).

Key Results

  • State-of-the-Art Accuracy: SS-MAE consistently outperformed existing baselines. On the Houston 2018 dataset, it achieved an Overall Accuracy (OA) of 92.43%, surpassing other advanced fusion models.
  • Robustness in Data-Scarce Scenarios: In few-shot learning tests (e.g., 16 samples per class), the pre-trained SS-MAE showed a significant performance lead over models trained from scratch, highlighting the effectiveness of its self-supervised representations.
  • Efficiency: Despite its dual-branch complexity, the model maintains a relatively low parameter count of approximately 4.4 million, though it does require substantial computational power (15.07 GFLOPs).

Conclusion

SS-MAE represents a significant step forward in applying self-supervised learning to multi-source remote sensing. By treating spectral information with the same importance as spatial structure, the framework unlocks the full potential of HSI, SAR, and LiDAR data. Future work will likely focus on optimizing computational efficiency and exploring more advanced cross-modal fusion strategies to further enhance classification precision.

Summary updated

BibTeX
@article{lin2023ss,
  author = {Junyan Lin and Feng Gao and Xiaocheng Shi and Junyu Dong and Qian Du},
  title = {{SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  doi = {10.1109/tgrs.2023.3331717},
}
Figure