P-MSDiff: Parallel Multi-Scale Diffusion for Remote Sensing Image Segmentation

Qi Zhang, Guohua Geng, Longquan Yan, Pengbo Zhou, Zhaodi Li, Kang Li, Qinglin LiuarXiv 2024 · DOI · arXiv

Introduction

Remote sensing (RS) image segmentation is a critical task for applications ranging from urban planning to environmental monitoring. However, imagery captured by drones and satellites is notoriously difficult to process due to high resolutions, complex backgrounds, and massive scale variations—where a single scene might contain both sprawling industrial complexes and tiny individual vehicles. While Diffusion Models have emerged as a powerful tool for generating high-quality segmentation masks through iterative denoising, they face significant hurdles in the RS domain, particularly regarding feature interpretation and computational efficiency.

The Challenges of Diffusion in Remote Sensing

Standard diffusion models typically rely on U-Net architectures with dense skip connections. In the context of RS, these connections can sometimes fail to convey semantic information efficiently across layers, making intermediate features difficult to interpret. Furthermore, to handle high-resolution images, researchers often employ "Linear Attention" to keep computational costs manageable. However, standard Linear Attention frequently suffers from "low-rank" issues, which makes it susceptible to noise and limits its ability to capture the intricate spatial details necessary for precise boundary delineation.

Methodology: P-MSDiff

To address these limitations, the authors introduce P-MSDiff, a parallel multi-scale architecture designed specifically for RS image segmentation.

Parallel Multi-Scale Architecture

Unlike traditional single-branch backbones, P-MSDiff utilizes parallel computational branches. The primary backbone, WNetformer, is augmented by a 1/2-scale parallel branch. This secondary branch operates at a lower resolution to learn noise patterns independently, which are then fused back into the main framework. This dual-path approach enhances the model's ability to synthesize semantic information across different levels of granularity, capturing both global context and local detail.

Overview of the P-MSDiff architecture featuring the WNetformer and the 1/2-Scale Parallel Branch.

Cross-Bridge Linear Attention (CBLA)

A core innovation of P-MSDiff is the Cross-Bridge Linear Attention (CBLA) mechanism. CBLA is a "plug-and-play" replacement for standard Linear Attention that introduces a semidefinite matrix—a "cross-bridge"—linked to the query ($Q$) into the dot-product computation of keys ($K$) and values ($V$).

Comparison between standard Linear Attention (a) and the proposed Cross-Bridge Linear Attention (b).

By incorporating this self-similarity bridge, the model can adapt queries dynamically and improve the weight distribution of multi-head attention. This effectively bypasses the low-rank bottleneck of traditional linear attention without incurring the quadratic computational explosion associated with standard Transformers.

Experimental Results

The model was rigorously evaluated on two prominent remote sensing benchmarks: UAVid and Vaihingen.

  • UAVid Dataset: This high-resolution (4K) drone dataset is particularly challenging due to its oblique perspective and small object classes like humans and moving cars. P-MSDiff achieved a 68.7% mIoU, a 1.60% improvement over the previous state-of-the-art (RNDiff). Notably, it showed dramatic gains in difficult categories, such as a +5.6% IoU improvement for "Human" detection.
  • Vaihingen Dataset: On this urban aerial dataset, the model reached 93.90% mIoU and a 98.93% F1-score.

Qualitative analysis reveals that P-MSDiff excels at maintaining sharp, geometric edges for man-made structures, avoiding the "rounded" or fragmented boundaries often produced by traditional CNN-based models.

Conclusion and Future Work

P-MSDiff demonstrates that parallel multi-scale processing and refined attention mechanisms can significantly boost the performance of diffusion models in remote sensing. While the iterative nature of diffusion models (requiring 128 denoising steps in this study) presents a challenge for real-time applications, the gains in boundary precision and small-object detection are substantial. Future research will focus on optimizing these models for speed and exploring their utility in hyperspectral imagery analysis.

Summary updated

BibTeX
@misc{zhang2024p,
  author = {Qi Zhang and Guohua Geng and Longquan Yan and Pengbo Zhou and Zhaodi Li and Kang Li and Qinglin Liu},
  title = {{P-MSDiff: Parallel Multi-Scale Diffusion for Remote Sensing Image Segmentation}},
  year = {2024},
  eprint = {2405.20443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.20443},
}
Figure