Multi-Class Segmentation from Aerial Views using Recursive Noise Diffusion

Benedikt Kolbeinsson, Krystian Mikolajczyk

Introduction

Semantic segmentation is a critical component for autonomous aerial systems, enabling drones to perceive and categorize their environment at a pixel level. However, aerial imagery presents unique challenges, including extreme scale variations, diverse viewpoints, and high environmental clutter. Traditional Convolutional Neural Networks (CNNs) often suffer from limited receptive fields, failing to capture the global context necessary for accurate building or infrastructure identification. While diffusion models have revolutionized generative AI, their application to discriminative tasks like multi-class segmentation has historically been limited by precision issues and computational overhead.

Methodology

This paper proposes a framework centered on Recursive Noise Diffusion, which adapts the iterative denoising process of diffusion models for high-precision semantic labeling.

Recursive Noise Diffusion

A common failure mode in standard diffusion training for segmentation is the model's tendency to rely on simple mathematical shortcuts (e.g., rounding) rather than deeply analyzing the input image. To counteract this, the authors introduce a recursive training pipeline. In this setup, the model is trained to progress through time steps recursively, forcing information propagation throughout the denoising process. This ensures the final segmentation is strictly conditioned on the input RGB image.

The recursive training pipeline where predicted segmentations are fed back into the noise function to ensure the model utilizes the conditioning RGB image.

Hierarchical Multi-scale Approach

To address the scale variation inherent in drone-captured data, the framework employs a coarse-to-fine strategy. The model first denoises a low-resolution version of the image to establish global structures. This initial prediction is then upscaled and refined at higher resolutions to capture intricate details, such as small vehicles or vegetation boundaries. The backbone of this system is WNetFormer, an architecture derived from UNetFormer that integrates the previous noisy segmentation map as a secondary input alongside the primary RGB photo.

The hierarchical multi-scale denoising process, starting from low-resolution global structures and scaling up to capture fine details.

Experimental Results

The proposed method was evaluated on two benchmark datasets: UAVid (high-resolution drone imagery with eight classes) and Vaihingen Buildings (focused on building segmentation).

  • State-of-the-Art Performance: On the Vaihingen dataset, the model achieved a 92.50% mean Intersection over Union (mIoU), surpassing previous benchmarks like SegDiff.
  • UAVid Benchmarking: On the more complex UAVid dataset, the model reached 67.1% mIoU. While slightly behind specialized architectures like DCDNet (72.4%), it significantly outperformed standard models such as UAVFormer and BANet.
  • Contextual Reasoning: Qualitatively, the model demonstrated an advanced ability to distinguish between static and moving vehicles by leveraging surrounding context, a task that often baffles single-pass networks.

Discussion and Conclusion

The iterative nature of diffusion models allows for self-correction, leading to sharper boundaries and better contextual integration than traditional one-shot CNNs. While inference speed remains a challenge due to the multiple denoising steps required, the authors demonstrated that skipping up to 50% of these steps during inference results in negligible accuracy loss. Future research will focus on optimizing hyperparameters and exploring knowledge distillation to enable real-time deployment on resource-constrained drone hardware.

Summary updated

BibTeX
@inproceedings{kolbeinsson2024multi,
  author = {Benedikt Kolbeinsson and Krystian Mikolajczyk},
  title = {{Multi-Class Segmentation from Aerial Views using Recursive Noise Diffusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00825},
}
Figure