MSSDF: Modality-Shared Self-supervised Distillation for High-Resolution Multi-modal Remote Sensing Image Learning

Tong Wang, Guanzhou Chen, Xiaodong Zhang, Chenxi Liu, Jiaqi Wang, Xiaoliang Tan, Wenchao Guo, Qingyuan Yang, Kaiqi Zhang

Introduction

Remote sensing has entered the era of "Ultra-High-Resolution" (UHR), enabling centimeter-level detail. However, two major bottlenecks persist: the extreme scarcity of manual labels for UHR imagery and the "modality gap" between visual RGB data and structural 3D data (like Digital Surface Models). Existing self-supervised learning (SSL) methods often struggle to capture complex cross-modal correlations or lose fine-grained structural details during the learning process.

To address these challenges, the authors propose MSSDF (Modality-Shared Self-supervised Distillation Framework), a foundation model designed to understand both visual and structural aspects of the Earth's surface without requiring massive labeled datasets. By combining an information-aware masking strategy with a student-teacher distillation architecture, it achieves state-of-the-art performance across 26 diverse tasks.

Methodology: The MSSDF Framework

MSSDF is built on a Student-Teacher paradigm using Vision Transformers (ViT). Unlike standard Masked Autoencoders (MAE) that use random masking, MSSDF employs several novel strategies to enhance learning from multi-modal data.

Comparison of MAE, Multi-MAE, and the proposed MSSDF architecture.

Key Components:

  • Information-Aware Adaptive Masking: The model calculates "information density" using gradient magnitude and local variance. It preserves semantically rich areas (e.g., building edges) while masking simpler regions (e.g., flat terrain) more aggressively.
  • Cross-Modal Masking: By selectively masking or substituting patches between RGB and 3D modalities, the framework forces the model to reason across data types to reconstruct missing information.
  • Multi-Task Objective: The learning process is guided by a four-part loss function: Reconstruction (MSE), Contrastive Alignment (CA) for cross-modal consistency, Feature Decorrelation (Linear HSIC) to reduce redundancy, and an Auxiliary Classification task to improve feature discriminability.

Datasets: The HR-Paris Benchmark

A core contribution of this work is the introduction of HR-Paris, a massive new resource for the remote sensing community. It contains 640,000 pixel-aligned pairs of Digital Orthophoto Maps (DOM) and Digital Surface Models (DSM) at a staggering 5cm spatial resolution. This dataset covers diverse terrains across more than 10 countries, providing a robust foundation for pre-training.

Results and Performance

The MSSDF model was evaluated across 15 established datasets and 26 diverse tasks, including scene classification, object detection, and monocular depth estimation. It consistently outperformed existing state-of-the-art (SOTA) models like Scale-MAE and OFA-L.

Radar chart comparing MSSDF performance against SOTA models across 26 tasks.

Key Findings:

  • Semantic Segmentation: On the Potsdam dataset, MSSDF achieved an mIoU of 78.30% using only 50% of training labels.
  • Change Detection: The model reached an mIoU of 47.51% on the SECOND dataset, a significant 3% improvement over previous methods.
  • Depth Estimation: On the US3D dataset, the model demonstrated a superior understanding of 3D geometry, reducing the Root Mean Square Error (RMSE) to 0.182.
  • Data Efficiency: MSSDF showed exceptional robustness in low-data regimes, maintaining high performance even with only 1% to 10% of downstream labels available.

Conclusion and Future Directions

MSSDF demonstrates that combining smart masking with feature distillation leads to more stable and semantically meaningful representations for UHR remote sensing. While the current focus is on RGB and DSM data, future work could expand to include Synthetic Aperture Radar (SAR) or thermal infrared bands. Additionally, exploring lightweight architectures for edge deployment on drones and satellites remains a key area for practical application.

Summary updated

BibTeX
@article{wang2026mssdf,
  author = {Tong Wang and Guanzhou Chen and Xiaodong Zhang and Chenxi Liu and Jiaqi Wang and Xiaoliang Tan and Wenchao Guo and Qingyuan Yang and Kaiqi Zhang},
  title = {{MSSDF: Modality-Shared Self-supervised Distillation for High-Resolution Multi-modal Remote Sensing Image Learning}},
  journal = {Information Fusion},
  year = {2026},
  doi = {10.1016/j.inffus.2025.104006},
}
Figure