FB-BEV: BEV Representation from Forward-Backward View Transformations

Zhiqi Li, Zhiding Yu, Wenhai Wang, Anima Anandkumar, Tong Lu, José M. Álvarez

Abstract

View Transformation Module (VTM), where transformations happen between multi-view image features and Bird-Eye-View (BEV) representation, is a crucial step in camera-based BEV perception systems. Currently, the two most prominent VTM paradigms are forward projection and backward projection. Forward projection, represented by Lift-Splat-Shoot, leads to sparsely projected BEV features without post-processing. Backward projection, with BEV-Former being an example, tends to generate false-positive BEV features from incorrect projections due to the lack of utilization on depth. To address the above limitations, we propose a novel forward-backward view transformation module. Our approach compensates for the deficiencies in both existing methods, allowing them to enhance each other to obtain higher quality BEV representations mutually. We instantiate the proposed module with FB-BEV, which achieves a new state-of-the-art result of 62.4% NDS on the nuScenes test set.

Introduction

In autonomous driving, "View Transformation" is the process of converting 2D camera images into a 3D Bird’s-Eye-View (BEV) representation. Current research typically relies on two main strategies, both of which have significant flaws. Forward Projection (e.g., LSS) "lifts" 2D pixels into 3D space based on predicted depth, but often results in sparse maps with "holes" or blank grids, especially for distant objects. Conversely, Backward Projection (e.g., BEVFormer) starts with 3D voxels and samples 2D features, but lacks depth guidance. This often leads to "ghost" objects and false positives as the model fetches identical features for every point along a projection ray.

FB-BEV aims to bridge this gap by creating a representation that is both dense and depth-accurate through a hybrid "Forward-Backward" approach.

Methodology

The FB-BEV pipeline introduces a three-step process designed to leverage the strengths of both projection paradigms:

  1. Forward VTM (F-VTM): The model generates an initial, sparse BEV map using forward projection. This provides a baseline that respects predicted depth.
  2. Foreground Region Proposal Network (FRPN): A lightweight CNN analyzes the sparse BEV map to predict a "foreground mask." This identifies specific Regions of Interest (RoI) in the BEV space that require refinement, improving computational efficiency.
  3. Depth-Aware Backward VTM (B-VTM): For the identified RoIs, the model performs a backward projection. Unlike standard methods, this stage incorporates a Depth Consistency mechanism to weight features based on how well a 3D point's position aligns with the predicted 2D depth distribution.
The FB-BEV architecture overview showing the integration of Forward and Backward VTMs.

Key Innovation: Depth Consistency

The core technical advancement in FB-BEV is the depth-consistency mechanism within the B-VTM. By bringing depth information into the backward projection process, the authors transform a standard feature fetch into a "smart" one. If a 3D point is sampled at a distance that contradicts the 2D depth estimation, the mechanism assigns a low weight to that projection. This effectively filters out the "smearing" artifacts and false positives that typically plague backward-projection models.

The depth-consistency mechanism used in B-VTM to filter false positives.

Results and Performance

FB-BEV was evaluated on the nuScenes dataset, a large-scale benchmark for autonomous driving. The model achieved a new state-of-the-art result of 62.4% NDS on the nuScenes test set, outperforming previous leaders. On the validation set, FB-BEV demonstrated significant gains over standalone forward (BEVDet) and backward (BEVFormer) baselines, improving NDS by 2.4% and 2.7% respectively. Qualitative results show that FB-BEV successfully eliminates "ray-shaped" artifacts, resulting in cleaner and more accurate 3D object detections.

Conclusion

By combining the depth-accuracy of forward projection with the density of backward projection, FB-BEV provides a robust solution for high-quality BEV representation. While the system still depends on the accuracy of its initial depth estimation, the mutual compensation between F-VTM and B-VTM sets a new standard for camera-based 3D perception. Future work may explore more sophisticated region proposal methods to further enhance efficiency in high-speed, long-distance scenarios.

Summary updated

Abstract

View Transformation Module (VTM), where transformations happen between multi-view image features and Bird-Eye-View (BEV) representation, is a crucial step in camera-based BEV perception systems. Currently, the two most prominent VTM paradigms are forward projection and backward projection. Forward projection, represented by Lift-Splat-Shoot, leads to sparsely projected BEV features without post-processing. Backward projection, with BEV-Former being an example, tends to generate false-positive BEV features from incorrect projections due to the lack of utilization on depth. To address the above limitations, we propose a novel forward-backward view transformation module. Our approach compensates for the deficiencies in both existing methods, allowing them to enhance each other to obtain higher quality BEV representations mutually. We instantiate the proposed module with FB-BEV, which achieves a new state-of-the-art result of 62.4% NDS on the nuScenes test set. Code and models are available at https://github.com/NVlabs/FB-BEV.
BibTeX
@inproceedings{li2023fb,
  author = {Zhiqi Li and Zhiding Yu and Wenhai Wang and Anima Anandkumar and Tong Lu and José M. Álvarez},
  title = {{FB-BEV: BEV Representation from Forward-Backward View Transformations}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=Ql22TepMNS},
}
Figure