DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception

Jiayu Zou, Zheng Zhu, Yun Ye, Xingang Wang

Abstract

BEV perception is of great importance in the field of autonomous driving, serving as the cornerstone of planning, controlling, and motion prediction. The quality of the BEV feature highly affects the performance of BEV perception. However, taking the noises in camera parameters and LiDAR scans into consideration, we usually obtain BEV representation with harmful noises. Diffusion models naturally have the ability to denoise noisy samples to the ideal data, which motivates us to utilize the diffusion model to get a better BEV representation. In this work, we propose an end-to-end framework, named DiffBEV, to exploit the potential of diffusion model to generate a more comprehensive BEV representation. To the best of our knowledge, we are the first to apply diffusion model to BEV perception. In practice, we design three types of conditions to guide the training of the diffusion model which denoises the coarse samples and refines the semantic feature in a progressive way. What's more, a cross-attention module is leveraged to fuse the context of BEV feature and the semantic content of conditional diffusion model. DiffBEV achieves a 25.9% mIoU on the nuScenes dataset, which is 6.2% higher than the best-performing existing approach. Quantitative and qualitative results on multiple benchmarks demonstrate the effectiveness of DiffBEV in BEV semantic segmentation and 3D object detection tasks.

Introduction

In autonomous driving, Bird’s Eye View (BEV) perception has emerged as the gold standard for spatial reasoning, enabling downstream tasks like motion planning and navigation. The process typically involves "lifting" 2D multi-camera images into a top-down 3D representation. However, this transformation is inherently noisy. Imperfect camera calibrations, sparse LiDAR data, and the mathematical approximations used in view transformers often lead to distorted BEV features.

Existing methods, such as Lift-Splat-Shoot (LSS), frequently produce "blurry" outputs where distinct objects merge into single blobs and static road elements like sidewalks or pedestrian crossings are misclassified. DiffBEV addresses these challenges by reframing BEV perception as a denoising problem. By adapting the iterative refinement capabilities of generative diffusion models, DiffBEV "polishes" coarse BEV features into high-fidelity representations with sharp boundaries.

The DiffBEV Framework

The core of DiffBEV is a Conditional Diffusion Probabilistic Model (DPM) designed to refine initial BEV features. Unlike standard diffusion models that generate data from pure Gaussian noise, DiffBEV uses the "coarse" output from a view transformer as a condition to guide the denoising process.

The DiffBEV framework architecture, featuring the Conditional Diffusion Probabilistic Model (DPM) and Cross-Attention fusion.

The architecture consists of three primary components:

  1. Conditioning and Iterative Refinement: The model takes a noised version of the BEV feature and progressively denoises it using a U-Net-based structure. The authors found that conditioning the model on depth distribution—a key source of spatial uncertainty—provided the most significant performance gains.
  2. Cross-Attention Fusion: To ensure the refined features remain grounded in the original sensor data and do not "hallucinate" details, a cross-attention module is employed. This module fuses the semantic content generated by the diffusion model with the original source BEV features.
  3. Task-Specific Decoders: The framework is end-to-end and modular, allowing the refined BEV features to be passed to various heads for tasks such as semantic segmentation or 3D object detection.

Experimental Results

DiffBEV was evaluated on industry-standard benchmarks, including nuScenes and KITTI (Raw, Odometry, and 3D Object).

Quantitative Performance

On the nuScenes dataset, DiffBEV achieved a state-of-the-art 25.9% mIoU for semantic segmentation, representing a 6.2% improvement over the previous best approach, LSS. The model showed particularly strong performance in difficult categories:

  • Cars: +11.6% improvement.
  • Pedestrian Crossings: +10.0% improvement.
  • Walkways: +6.7% improvement.

For 3D object detection, DiffBEV provided a ~1% improvement in the NuScenes Detection Score (NDS) over established baselines like BEVDet, demonstrating the versatility of the refined features.

Qualitative Analysis

The visual results highlight DiffBEV's ability to solve the "blob" problem. While traditional models often struggle to separate adjacent vehicles, DiffBEV’s iterative denoising process allows it to resolve fine-grained details and produce much sharper object boundaries.

Qualitative comparison of BEV semantic segmentation results on the nuScenes dataset. DiffBEV produces significantly sharper boundaries and more accurate object separation compared to baselines.

Conclusion and Future Work

DiffBEV demonstrates that diffusion models are not just powerful generators but also exceptional denoisers for spatial perception tasks. By treating the distortions in BEV features as noise to be removed, the framework achieves significant gains in both accuracy and visual clarity.

However, the iterative nature of diffusion models introduces computational latency. While the authors optimized the U-Net architecture to manage GFLOPs, real-time deployment on automotive hardware remains a challenge. Future research will focus on faster sampling methods, such as DDIM, and extending the framework to temporal tasks like motion prediction and occupancy grid mapping.

Summary updated

Abstract

BEV perception is of great importance in the field of autonomous driving, serving as the cornerstone of planning, controlling, and motion prediction. The quality of the BEV feature highly affects the performance of BEV perception. However, taking the noises in camera parameters and LiDAR scans into consideration, we usually obtain BEV representation with harmful noises. Diffusion models naturally have the ability to denoise noisy samples to the ideal data, which motivates us to utilize the diffusion model to get a better BEV representation. In this work, we propose an end-to-end framework, named DiffBEV, to exploit the potential of diffusion model to generate a more comprehensive BEV representation. To the best of our knowledge, we are the first to apply diffusion model to BEV perception. In practice, we design three types of conditions to guide the training of the diffusion model which denoises the coarse samples and refines the semantic feature in a progressive way. What's more, a cross-attention module is leveraged to fuse the context of BEV feature and the semantic content of conditional diffusion model. DiffBEV achieves a 25.9% mIoU on the nuScenes dataset, which is 6.2% higher than the best-performing existing approach. Quantitative and qualitative results on multiple benchmarks demonstrate the effectiveness of DiffBEV in BEV semantic segmentation and 3D object detection tasks. The code will be available soon.
BibTeX
@inproceedings{zou2024diffbev,
  author = {Jiayu Zou and Zheng Zhu and Yun Ye and Xingang Wang},
  title = {{DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=covrUg1s2h},
}
Figure