DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception

Jiayu Zou, Kun Tian, Zheng Zhu, Yun Ye, Xingang Wang

TL;DR

This paper introduces DiffBEV, a novel framework that leverages conditional diffusion models to significantly improve the quality of Bird's Eye View (BEV) representations for autonomous driving. By progressively refining noisy BEV features and fusing them with original features via cross-attention, DiffBEV achieves state-of-the-art performance in BEV semantic segmentation and 3D object detection.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on Bird's Eye View (BEV) perception to understand the surrounding environment for tasks like planning and control. However, existing methods often produce BEV features that are noisy and imprecise due to factors like inaccurate camera parameters and sensor noise. This degradation in BEV feature quality directly impacts the performance of downstream tasks, leading to issues like ambiguous object boundaries and rough segmentation of drivable areas. The paper aims to address this by generating a more comprehensive and accurate BEV representation.

Methods

DiffBEV introduces a novel approach by applying conditional diffusion models (DPMs) to BEV perception. The core idea is to use DPMs, known for their ability to denoise and generate high-quality data, to refine the noisy BEV features obtained from standard view transformers.

Here's a breakdown of the key components:

  • Conditional Diffusion Model: The DPM is conditioned on the original BEV feature (from a view transformer) and/or semantic features derived from depth distributions. This conditioning guides the diffusion process to denoise and enhance the BEV representation progressively.
  • Progressive Refinement: The diffusion model iteratively denoises a noisy BEV feature, gradually refining its semantic content and spatial details.
  • Cross-Attention Module: A novel cross-attention mechanism is employed to fuse the refined BEV features from the diffusion model with the original BEV features. This module adaptively learns the relationships between the generated and source features, leading to a more precise and compact perception result.
  • End-to-End Framework: DiffBEV is designed as an end-to-end trainable framework, meaning it can be optimized directly for downstream tasks without requiring separate pre-training stages for the diffusion model.
  • Flexible Task-Specific Decoders: The framework is modular, allowing for easy integration of different task-specific decoders for tasks like semantic segmentation and 3D object detection.

Datasets

The paper evaluates DiffBEV on several standard autonomous driving benchmarks:

  • nuScenes: Used for both BEV semantic segmentation and 3D object detection.
  • KITTI Raw: Used for BEV semantic segmentation of static road layouts.
  • KITTI Odometry: Used for BEV semantic segmentation of static road layouts.
  • KITTI 3D Object: Used for BEV semantic segmentation of dynamic vehicles.

Key Insights and Ideas

  • Diffusion Models for BEV Refinement: The paper pioneers the application of diffusion models for enhancing BEV perception, demonstrating their effectiveness in denoising and enriching noisy BEV features.
  • Conditioning is Key: The choice of conditioning for the diffusion model significantly impacts performance. Using semantic features derived from depth distributions (FS-BEV) proved most effective for improving segmentation accuracy.
  • Cross-Attention for Feature Fusion: The proposed cross-attention module is crucial for effectively integrating the refined diffusion model output with the original BEV features, leading to better downstream task performance.
  • End-to-End Learning Advantage: By avoiding separate pre-training, DiffBEV simplifies the training pipeline and allows for direct optimization of the diffusion model for specific perception tasks.
  • Generality Across View Transformers: DiffBEV's effectiveness is not tied to a single view transformer architecture, showing consistent improvements when integrated with various existing methods like VPN, PYVA, and PON.

Key Results

DiffBEV demonstrates significant improvements across multiple tasks and datasets:

  • BEV Semantic Segmentation (nuScenes): Achieved a mean Intersection over Union (mIoU) of 25.9%, which is 6.2% higher than the previous best-performing approach. It showed substantial gains in specific classes like drivable area (+5.0% IoU), pedestrian crossing (+10.0% IoU), walkway (+6.7% IoU), and car (+11.6% IoU).
  • 3D Object Detection (nuScenes): Improved the NuScenes Detection Score (NDS) by approximately 1% when integrated with BEVDet and by 1% when integrated with BEVDet4D, without requiring additional bells and whistles.
  • KITTI Benchmarks: Showed state-of-the-art performance on KITTI Raw, KITTI Odometry, and KITTI 3D Object datasets for semantic segmentation.

Qualitatively, DiffBEV produces clearer object boundaries and more accurate segmentation of drivable areas, effectively resolving issues like connected vehicle boundaries and misclassified backgrounds seen in prior methods.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, diffusion models are generally computationally intensive. The paper mentions a trade-off between self-attention and convolutional encoding mechanisms for noisy BEV samples, suggesting ongoing exploration of efficiency.
  • Future Work: The authors express a desire to further explore the potential of DiffBEV and broaden its application range to more perception tasks. They also highlight the generality of their approach by showing its effectiveness with various view transformers.

Summary updated

Abstract

BEV perception is of great importance in the field of autonomous driving, serving as the cornerstone of planning, controlling, and motion prediction. The quality of the BEV feature highly affects the performance of BEV perception. However, taking the noises in camera parameters and LiDAR scans into consideration, we usually obtain BEV representation with harmful noises. Diffusion models naturally have the ability to denoise noisy samples to the ideal data, which motivates us to utilize the diffusion model to get a better BEV representation. In this work, we propose an end-to-end framework, named DiffBEV, to exploit the potential of diffusion model to generate a more comprehensive BEV representation. To the best of our knowledge, we are the first to apply diffusion model to BEV perception. In practice, we design three types of conditions to guide the training of the diffusion model which denoises the coarse samples and refines the semantic feature in a progressive way. What's more, a cross-attention module is leveraged to fuse the context of BEV feature and the semantic content of conditional diffusion model. DiffBEV achieves a 25.9% mIoU on the nuScenes dataset, which is 6.2% higher than the best-performing existing approach. Quantitative and qualitative results on multiple benchmarks demonstrate the effectiveness of DiffBEV in BEV semantic segmentation and 3D object detection tasks.
BibTeX
@inproceedings{zou2024diffbev,
  author = {Jiayu Zou and Kun Tian and Zheng Zhu and Yun Ye and Xingang Wang},
  title = {{DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=covrUg1s2h},
}
Figure