MSSDF: Modality-shared self-supervised distillation for high-resolution multi-modal remote sensing image learning

Tong Wang, Guanzhou Chen, Xiaodong Zhang, Chenxi Liu, Jiaqi Wang, Xiaoliang Tan, Wenchao Guo, Qingyuan Yang, Kaiqi Zhang

TL;DR

MSSDF is a novel self-supervised distillation framework designed to master ultra-high-resolution (UHR) multi-modal remote sensing data. By combining an information-aware masking strategy with a student-teacher distillation architecture, it achieves state-of-the-art performance across 26 diverse tasks, supported by a massive new 5cm-resolution dataset called HR-Paris.

What Problem Does This Paper Address?

Remote sensing has entered the era of "Ultra-High-Resolution" (UHR), where we can see details at the centimeter level. However, two major bottlenecks remain:

  1. The Label Scarcity: Manually labeling UHR imagery for tasks like building extraction or change detection is incredibly expensive and slow.
  2. Modality Gap: Modern sensors provide more than just RGB images; they provide 3D data (Digital Surface Models or DSMs) and multi-spectral bands. Existing self-supervised learning (SSL) methods often fail to capture the complex correlations between these different "languages" of data or lose fine-grained structural details during the learning process.

The authors aim to create a foundation model that understands both the visual (RGB) and structural (3D/DSM) aspects of the Earth's surface without needing millions of human-made labels.

Methods

The authors introduce MSSDF (Modality-Shared Self-supervised Distillation Framework). Its architecture is built on a Student-Teacher paradigm using Vision Transformers (ViT):

  • Information-Aware Adaptive Masking: Unlike standard models that mask image patches randomly, MSSDF calculates the "information density" of patches using gradient magnitude and local variance. It keeps semantically rich areas (like building edges) and masks simpler areas (like flat grass) more aggressively.
  • Cross-Modal Masking: To force the model to understand how RGB and 3D data relate, the framework selectively masks or substitutes patches between modalities. This encourages the model to "reason" across modalities to reconstruct missing information.
  • Multi-Task Objective: The model isn't just trying to "fill in the blanks." It uses a four-part loss function:
  • Reconstruction (MSE): Predicting missing features.
  • Contrastive Alignment (CA): Ensuring corresponding patches in different modalities are mathematically close.
  • Feature Decorrelation (Linear HSIC): Preventing the model from learning redundant information across modalities.
  • Auxiliary Classification: A simple task where the model identifies which modality a patch belongs to, improving feature discriminability.

Datasets

The paper introduces a significant new resource: HR-Paris.

  • HR-Paris: Contains 640,000 pixel-aligned pairs of DOM (Digital Orthophoto Maps) and DSM (Digital Surface Models) at a staggering 5cm spatial resolution. It covers diverse terrains across 10+ countries.
  • Evaluation: The model was tested on 15 established datasets (including AID, NWPU, Potsdam, Vaihingen, and DOTA) covering 26 different tasks such as scene classification, object detection, and monocular depth estimation.

Key Insights and Ideas

  • Smart Masking Matters: Random masking is inefficient for remote sensing. By focusing the model's "attention" on high-contrast, high-variance areas during pre-training, the model learns structural features much faster.
  • Distillation over Reconstruction: Instead of trying to reconstruct raw pixels (which can be noisy), the student model tries to reconstruct the features generated by a teacher model (updated via Exponential Moving Average). This leads to more stable and semantically meaningful representations.
  • Redundancy is the Enemy: Multi-modal data often contains overlapping information. The use of Linear HSIC for decorrelation ensures that the model extracts unique, complementary features from each modality rather than just learning the same thing twice.

Key Results

MSSDF achieved state-of-the-art (SOTA) results across the board:

  • Semantic Segmentation: On the Potsdam dataset, the model achieved an mIoU of 78.30% using only 50% of the training labels, outperforming existing SOTA models like Scale-MAE and OFA-L.
  • Change Detection: On the SECOND dataset, it reached an mIoU of 47.51%, a 3% improvement over the previous best method.
  • Depth Estimation: On the US3D dataset, the RMSE (Root Mean Square Error) was reduced to 0.182, demonstrating a superior understanding of 3D geometry from 2D images.
  • Few-Shot Learning: The model showed exceptional robustness in "low-data" regimes, maintaining high accuracy even when only 1% to 10% of downstream labels were available.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Modality Expansion: The current work focuses heavily on RGB and DSM. Future iterations could incorporate Synthetic Aperture Radar (SAR) or thermal infrared data.
  • Edge Deployment: The use of large ViT backbones (ViT-L) provides great accuracy but is computationally heavy. The authors suggest exploring lightweight architectures for real-time deployment on drones or satellites.
  • Broader Applications: Moving beyond standard benchmarks to real-world applications like immediate disaster response or autonomous urban planning.

Summary updated

BibTeX
@article{wang2026mssdf,
  author = {Tong Wang and Guanzhou Chen and Xiaodong Zhang and Chenxi Liu and Jiaqi Wang and Xiaoliang Tan and Wenchao Guo and Qingyuan Yang and Kaiqi Zhang},
  title = {{MSSDF: Modality-shared self-supervised distillation for high-resolution multi-modal remote sensing image learning}},
  journal = {Information Fusion},
  year = {2026},
  doi = {10.1016/j.inffus.2025.104006},
}
Figure