Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing

Maofeng Tang, Andrei Cozma, Konstantinos Georgiou, Hairong Qi

Introduction

Remote sensing (RS) imagery presents a unique challenge due to its multi-scale nature. Hardware constraints often force a trade-off between spatial resolution and spectral or temporal coverage, resulting in datasets with varying Ground Sample Distances (GSD). This variability leads to two primary hurdles: the scarcity of labeled data across all scales and the difficulty of aligning features from images captured at different resolutions. Conventional models often ignore these spatial scale characteristics, leading to sub-optimal feature extraction. To address this, the authors propose Cross-Scale MAE, a self-supervised framework designed to learn scale-invariant features without requiring pre-aligned multi-scale pairs.

Methodology

The Cross-Scale MAE framework is built upon the Masked Auto-Encoder (MAE) architecture, which learns by reconstructing masked image patches. The model introduces three key innovations to handle multi-resolution data:

  1. Scale Augmentation: Instead of requiring real-world aligned image pairs, the framework synthetically generates two versions of a single image at different scales through random cropping and resizing. This forces the model to learn features that persist across different "zoom levels."
  2. Cross-Scale Consistency (Encoder): A contrastive loss (InfoNCE) is applied at the encoder level. It pushes the representations of the same site at different scales closer together in the feature space while pushing representations of different sites further apart. This ensures structural consistency, allowing the model to identify the same objects regardless of resolution.
  3. Cross-Scale Prediction (Decoder): The decoder implements a cross-prediction mechanism where embeddings from a low-resolution view are used to predict the embeddings of the high-resolution version. This ensures semantic consistency, helping the model understand what those objects are across resolutions.
The Cross-Scale MAE training pipeline, featuring dual-branch encoders and decoders optimized with contrastive and reconstruction losses.

To make the model accessible for researchers with limited resources, the authors leveraged the xFormers library, optimizing memory and speed to allow pre-training on a single GPU without sacrificing performance.

Experimental Evaluation

The model was pre-trained on the fMoW-RGB (Functional Map of the World) dataset, containing over 363k images with varying resolutions. Evaluation was conducted on several standard benchmarks: RESISC45, WHU-RS19, UC Merced, and EuroSAT for classification, and Potsdam and Vaihingen for segmentation.

Classification Performance and Robustness

Cross-Scale MAE consistently outperformed baselines such as SatMAE and the previous state-of-the-art, Scale-MAE. On the RESISC45 dataset, it achieved a KNN classification accuracy of 75.6%, significantly higher than SatMAE (66.3%) and Scale-MAE (70.0%).

A key finding of the study is the model's robustness to extreme scaling. Even when images were downscaled to 12.5% of their original size—a range the model did not encounter during training—Cross-Scale MAE maintained a significant performance lead over its peers.

KNN classification accuracy across various scale ratios, highlighting the performance stability of Cross-Scale MAE compared to baselines.

Downstream Segmentation

In semantic segmentation tasks on the Potsdam dataset, the model achieved a Mean Intersection over Union (mIoU) of 76.17%, outperforming Scale-MAE (75.07%). This demonstrates that the scale-invariant features learned during pre-training translate effectively to dense prediction tasks.

Discussion and Conclusion

The success of Cross-Scale MAE suggests that simulated multi-scale augmentation is a highly effective substitute for expensive, perfectly aligned multi-sensor data. One of the most powerful insights is that simple resizing combined with the right loss functions can bridge the GSD gap.

While highly effective for spatial scaling, the authors acknowledge limitations regarding spectral and temporal gaps. Future work aims to incorporate a more holistic multi-spectral perspective and investigate more refined contrastive strategies to further improve the model's ability to handle the diverse nature of satellite imagery.

Summary updated

Abstract

Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning problem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE).During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned representations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.
BibTeX
@inproceedings{tang2023cross,
  author = {Maofeng Tang and Andrei Cozma and Konstantinos Georgiou and Hairong Qi},
  title = {{Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=5oEVdOd6TV},
}
Figure