CVFNet: Real-time 3D Object Detection by Learning Cross View Features

Jiaqi Gu, Zhiyu Xiang, Pan Zhao, Tingming Bai, Lingxuan Wang, Zhiyuan Zhang

Introduction

In the realm of autonomous driving, 3D object detection using LiDAR point clouds is a critical task that demands both high precision and low latency. However, existing methodologies often face a significant "speed vs. accuracy" trade-off. Voxel-based methods offer high accuracy but are computationally expensive due to 3D convolutions. Point-based methods preserve raw geometry but suffer from slow neighbor-searching operations. Conversely, view-based methods, which project 3D data into 2D images, are exceptionally fast but typically lose vital spatial information, leading to degraded performance.

CVFNet (Cross View Feature Network) is a real-time, single-stage 3D object detector designed to bridge this gap. By progressively fusing features across three distinct views—Point, Range, and Bird’s Eye View (BEV)—CVFNet achieves high-tier accuracy while maintaining the efficiency of 2D convolutions.

Methodology

The core of CVFNet lies in its progressive cross-view feature learning pipeline, which transitions data through a "Range → Point → BEV" sequence. This approach ensures that the network captures both dense textural details and precise geometric structures.

1. Point-Range Fusion Module (PRNet)

Rather than selecting a single projection, CVFNet processes the Range View (the native spherical projection of LiDAR) and the Point View (raw 3D coordinates) in parallel. The PRNet utilizes a bilateral aggregation strategy to fuse these features at multiple stages. This multi-stage fusion prevents the "information collapse" common in single-plane projections, allowing the model to leverage the dense context of range images alongside the geometric fidelity of raw points.

2. Slice Pillar Transformation

To transition from point-wise features to a Bird’s Eye View (BEV) for final detection, the authors introduced the "Slice Pillar" transformation. Traditional "Pillars" often collapse all height information into a single feature vector, losing vertical granularity. Slice Pillars instead utilize fast "scatter" operations to create a voxel-like volume. This volume is then processed using 1x1 and 2D convolutions, effectively treating the height dimension as a feature channel. This allows the network to maintain 3D spatial relationships without the heavy computational tax of 3D convolutions.

3. Sparse Pillar Head

Standard detectors often perform computations across every grid cell in a BEV map, including empty space. CVFNet introduces a Sparse Pillar Head that filters out empty pillars, focusing computational resources only on non-empty grids. This optimization balances the ratio of positive to negative samples and significantly accelerates inference speeds.

Experimental Results

CVFNet was evaluated on two industry-standard benchmarks: KITTI and NuScenes. The results demonstrate that the model is highly competitive with state-of-the-art methods while offering superior efficiency.

  • Efficiency: On an NVIDIA 2080Ti GPU, CVFNet achieved an inference speed of 35.6 Hz on the KITTI dataset and 16.4 Hz on the larger NuScenes dataset.
  • Accuracy: On the KITTI benchmark, CVFNet outperformed the widely-used PointPillars baseline by 6.17% in 3D Average Precision (AP) for the "Car" category.
  • Performance: The model achieved top-tier results among view-based detectors and remained competitive with significantly slower voxel and point-based architectures.

Conclusion and Future Work

CVFNet demonstrates that 3D convolutions are not a prerequisite for high-quality 3D detection. By intelligently managing transformations between views and utilizing efficient 2D convolutions, CVFNet provides a robust solution for real-time applications.

Looking forward, the authors suggest that integrating additional modalities, such as RGB camera images, could further enhance the system. By adding a "Camera View" to the progressive fusion pipeline, the model could leverage color and fine-grained texture to improve detection in complex urban environments. Additionally, exploring more advanced 2D backbones may yield further improvements in precision without sacrificing the real-time capabilities of the network.

Summary updated

Abstract

In recent years 3D object detection from LiDAR point clouds has made great progress thanks to the development of deep learning technologies. Although voxel or point based methods are popular in 3D object detection, they usually involve time-consuming operations such as 3D convolutions on voxels or ball query among points, making the resulting network inappropriate for time critical applications. On the other hand, 2D view-based methods feature high computing efficiency while usually obtaining inferior performance than the voxel or point based methods. In this work, we present a real-time view-based single stage 3D object detector, namely CVFNet to fulfill this task. To strengthen the cross-view feature learning under the condition of demanding efficiency, our framework extracts the features of different views and fuses them in an efficient progressive way. We first propose a novel Point-Range feature fusion module that deeply integrates point and range view features in multiple stages. Then, a special Slice Pillar is designed to well maintain the 3D geometry when transforming the obtained deep point-view features into bird's eye view. To better balance the ratio of samples, a sparse pillar detection head is presented to focus the detection on the nonempty grids. We conduct experiments on the popular KITTI and NuScenes benchmark, and state-of-the-art performances are achieved in terms of both accuracy and speed.
BibTeX
@inproceedings{gu2022cvfnet,
  author = {Jiaqi Gu and Zhiyu Xiang and Pan Zhao and Tingming Bai and Lingxuan Wang and Zhiyuan Zhang},
  title = {{CVFNet: Real-time 3D Object Detection by Learning Cross View Features}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=C3NjBd7m7M},
}
Figure