SS-MAE: Spatial-Spectral Masked Autoencoder for Multisource Remote Sensing Image Classification

Junyan Lin, Feng Gao, Xiaochen Shi, Junyu Dong, Qian Du

TL;DR

This paper introduces SS-MAE, a novel self-supervised learning framework for classifying remote sensing images from multiple sources (like hyperspectral, LiDAR, and SAR). By simultaneously masking and reconstructing both spatial patches and spectral channels, SS-MAE effectively learns rich representations, outperforming existing methods on benchmark datasets.

What Problem Does This Paper Address?

Remote sensing data, such as hyperspectral (HSI), LiDAR, and SAR, offers complementary information for land cover classification. However, effectively fusing these diverse data types remains a challenge. Existing deep learning methods often require large amounts of labeled data, which are expensive and time-consuming to acquire. While masked autoencoders (MAEs) have shown promise in self-supervised learning for general image understanding, they often focus on spatial features and neglect spectral information, or lose local details due to their reliance on Transformers. This paper addresses the need for a robust, self-supervised method that can effectively leverage both spatial and spectral information from multi-source remote sensing data, while also capturing both global and local features.

Methods

The core of this work is the Spatial-Spectral Masked Auto-Encoder (SS-MAE). It operates in two main stages:

  • Pre-training Stage: This is the self-supervised learning phase.
  • Dual-Branch Masking: SS-MAE employs two parallel branches. The spatial-wise branch masks random patches of the input image and reconstructs the missing pixels. The spectral-wise branch masks random spectral channels and reconstructs the missing channels. This dual approach ensures that both spatial and spectral representations are learned.
  • Transformer and CNN Hybrid: The model uses a Transformer-based encoder-decoder architecture for learning global feature dependencies. To complement this and capture local features (like edges and textures), two lightweight Convolutional Neural Networks (CNNs) with inverted residual blocks (IRBs) are incorporated.
  • Reconstruction Target: The model is trained to reconstruct the original input from the masked versions, minimizing the mean squared error between the predicted and original data.
  • Training and Test Stage:
    • The pre-trained encoders are used to extract features.
  • These features are then concatenated with features extracted by the lightweight CNNs (for local information).
  • A classifier is trained on these combined features for the final land cover classification task.

Datasets

The proposed SS-MAE was evaluated on three publicly available multi-source remote sensing datasets:

  • Berlin Dataset: Hyperspectral (simulated EnMAP) and SAR (Sentinel-1 dual-Pol) data.
  • Augsburg Dataset: Hyperspectral (HySpex) and SAR (Sentinel-1) data.
  • Houston 2018 Dataset: Hyperspectral and LiDAR data.

The paper provides detailed tables enumerating the number of training and testing samples for each class within these datasets.

Key Insights and Ideas

  • Spectral Information is Crucial: The study highlights that spectral attention is critical for hyperspectral image interpretation, often yielding better results than spatial attention alone.
  • Balancing Global and Local Features: The combination of Transformer (for global context) and CNNs (for local details) is essential for robust feature representation in remote sensing.
  • Effective Self-Supervised Pre-training: SS-MAE demonstrates that masked autoencoding, when applied to both spatial and spectral dimensions, can learn powerful representations from unlabeled data, significantly improving downstream classification performance.
  • Data-Specific Parameter Tuning: Optimal parameters for masking ratios and pre-training sample sizes vary across datasets, suggesting that dataset characteristics influence model training.

Key Results

  • State-of-the-Art Performance: SS-MAE consistently outperformed several state-of-the-art methods across the three benchmark datasets, achieving higher Overall Accuracy (OA), Average Accuracy (AA), and Kappa values. For instance, on the Berlin dataset, SS-MAE achieved an OA of 73.43%, surpassing other methods by a notable margin.
  • Pre-training Benefits: The pre-trained SS-MAE model showed significant improvements over the non-pre-trained version (SS-MAE*), especially in challenging classes with similar spectral responses.
  • Robustness: The model demonstrated strong performance even with limited labeled data (few-shot classification experiments).
  • Efficiency: While not the fastest, the model's computational complexity (parameters and GFLOPS) was competitive, especially considering its unsupervised learning approach.

Limitations and Future Work

  • Parameter Sensitivity: The performance of SS-MAE is influenced by parameter choices like masking ratio and input patch size, requiring careful tuning for different datasets.
  • Computational Cost: While efficient for an unsupervised method, the Transformer architecture can still be computationally intensive.
  • Future Directions: The authors plan to explore more diverse cross-modal feature fusion strategies and conduct further experiments on masked image modeling for multi-source remote sensing data classification.

Summary updated

BibTeX
@article{lin2023ss,
  author = {Junyan Lin and Feng Gao and Xiaochen Shi and Junyu Dong and Qian Du},
  title = {{SS-MAE: Spatial-Spectral Masked Autoencoder for Multisource Remote Sensing Image Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  doi = {10.1109/tgrs.2023.3331717},
}
Figure