Multi-Class Segmentation from Aerial Views using Recursive Noise Diffusion

Benedikt Kolbeinsson, Krystian Mikolajczyk

Introduction

Researchers from Imperial College London have adapted diffusion models—the technology behind generative AI like DALL-E—for the precise task of multi-class semantic segmentation in aerial imagery. By introducing recursive denoising and a hierarchical multi-scale approach, their framework achieves state-of-the-art performance on building segmentation and effectively handles the extreme scale variations common in drone-captured data.

The Challenge of Aerial Segmentation

Semantic segmentation, the process of labeling every pixel in an image (e.g., "road," "tree," or "building"), is vital for autonomous drone navigation. However, aerial views present unique difficulties for standard AI models. Drones capture scenes with extreme scale variations—such as a tiny car next to a massive warehouse—and diverse viewpoints that traditional convolutional neural networks (CNNs) often struggle to process holistically.

While diffusion models have revolutionized image generation, applying them to multi-class segmentation is complex. Most existing diffusion-based segmentation methods are limited to binary tasks or rely on pre-trained generative features. This research introduces an end-to-end diffusion framework that can handle multiple categories simultaneously in high-resolution aerial scenes.

Methodology: Recursive Noise Diffusion

The proposed framework, Recursive Noise Diffusion (RND), builds on the standard Denoising Diffusion Probabilistic Model (DDPM) but introduces two major innovations:

1. Recursive Denoising

In standard diffusion training, models learn to denoise a single, randomly chosen noise level. The authors found this leads to overfitting in segmentation tasks because the model learns "trivial" shortcuts rather than a deep understanding of the scene. Their recursive approach involves training the model to progress through time steps sequentially, allowing errors from one step to propagate to the next. This forces the model to learn how to use the conditioning RGB image more effectively to "correct" its predictions.

The recursive training pipeline allows the model to learn from its own previous denoising errors.

2. Hierarchical Multi-Scale Diffusion

To handle scale issues, the model starts by denoising a low-resolution version of the segmentation map. This allows it to capture large-scale structures and global context. Halfway through the process, it upscales the prediction and finishes denoising at full resolution to capture fine details and small objects.

The multi-scale denoising process starts at a lower resolution to capture global context before scaling up for fine details.

For the underlying architecture, the researchers modified a transformer-based model called UNetFormer into WNetFormer, which accepts both the original image and the current noisy segmentation map as inputs.

Key Results and Insights

The model was evaluated on two challenging aerial benchmarks: UAVid (a high-resolution 4K drone dataset with 8 classes) and Vaihingen Buildings (focused on building footprint extraction).

  • State-of-the-Art (SOTA): On the Vaihingen Buildings dataset, the model achieved a mean Intersection over Union (mIoU) of 92.50%, outperforming previous specialized methods like SegDiff.
  • Competitive Performance: On the complex UAVid dataset, it reached 67.1 mIoU, performing comparably to or better than several recent high-end transformer models.
  • Efficiency Gains: While diffusion is usually slow, the authors showed that recursive training allows for skipping up to 50% of inference steps with almost no loss in accuracy. They successfully ran inference with as few as 16–32 steps, whereas generative models often require hundreds.

Conclusion and Future Directions

The primary hurdle remains inference speed, as diffusion models are computationally heavier than standard "one-shot" CNNs or Transformers. Additionally, the "Human" class in the UAVid dataset remains a challenge due to the extreme rarity and small size of people in 4K aerial shots. Future work will likely focus on knowledge distillation and optimizing hyperparameters to bridge the gap between high-accuracy diffusion and the real-time requirements of drone hardware.

Summary updated

Abstract

Semantic segmentation from aerial views is a crucial task for autonomous drones, as they rely on precise and accurate segmentation to navigate safely and efficiently. However, aerial images present unique challenges such as diverse viewpoints, extreme scale variations, and high scene complexity. In this paper, we propose an end-to-end multi-class semantic segmentation diffusion model that addresses these challenges. We introduce recursive denoising to allow information to propagate through the denoising process, as well as a hierarchical multi-scale approach that complements the diffusion process. Our method achieves promising results on the UAVid dataset and state-of-the-art performance on the Vaihingen Building segmentation benchmark. Being the first iteration of this method, it shows great promise for future improvements.
BibTeX
@inproceedings{kolbeinsson2024multi,
  author = {Benedikt Kolbeinsson and Krystian Mikolajczyk},
  title = {{Multi-Class Segmentation from Aerial Views using Recursive Noise Diffusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00825},
}
Figure