JPV-Net: Joint Point-Voxel Representations for Accurate 3D Object Detection

Nan Song, Tianyuan Jiang, Jian YaoAAAI 2022 · OpenReview

Abstract

Voxel and point representations are widely applied in recent 3D object detection tasks from LiDAR point clouds. Voxel representations contribute to efficiently and rapidly locating objects, whereas point representations are capable of describing intra-object spatial relationship for detection refinement. In this work, we aim to exploit the strengths of both two representations, and present a novel two-stage detector, named Joint Point-Voxel Network (JPV-Net). Specifically, our framework is equipped with a Dual Encoders-Fusion Decoder, which consists of the dual encoders to extract voxel features of sketchy 3D scenes and point features rich in geometric context, respectively, and the Feature Propagation Fusion (FP-Fusion) decoder to attentively fuse them from coarse to fine. By making use of the advantages of these features, the refinement network can effectively eliminate false detection and provide better accuracy. Besides, to further develop the perception characteristics of voxel CNN and point backbone, we design two novel intersection-over-union (IoU) estimation modules for proposal generation and refinement, both of which can alleviate the misalignment between the localization and the classification confidence. Extensive experiments on the KITTI dataset and ONCE dataset demonstrate that our proposed JPV-Net outperforms other state-of-the-art methods with remarkable margins.

Introduction

In the field of autonomous driving, 3D object detection from LiDAR point clouds is a critical task. Current research typically follows one of two paths: voxel-based methods or point-based methods. Voxel-based methods discretize point clouds into regular 3D grids, enabling fast processing and effective global localization. However, this "voxelization" process is inherently lossy, sacrificing fine-grained geometric details and resulting in "sketchy" object representations. Conversely, point-based methods process raw points directly, preserving intricate shapes but at a high computational cost and with limited global context.

Hybrid methods attempt to combine these representations but often suffer from misalignment: a detector might show high classification confidence for a proposal that has poor localization accuracy. JPV-Net addresses these challenges by bridging the gap between voxel and point representations while introducing novel IoU-aware modules to synchronize confidence with precision.

Methodology

JPV-Net is a two-stage detector designed to leverage the complementary strengths of voxels and points through a sophisticated architecture.

The JPV-Net architecture featuring dual encoders, an FP-Fusion decoder, and IoU-aware modules for proposal generation and refinement.

The core of the system is the Dual Encoders-Fusion Decoder. One branch utilizes a standard voxel CNN to provide a coarse sketch of the 3D scene, while a parallel branch employs a PointNet++ backbone to extract raw geometric context. These features are integrated via a Trilinear Set Abstraction (SA) module and an FP-Fusion decoder. The FP-Fusion module uses an attention mechanism to dynamically weight voxel and point features across multiple scales, ensuring the most relevant information is prioritized.

To improve localization, JPV-Net incorporates two IoU-aware modules:

  • BEV-IoU-aware Proposal Generation: This stage predicts the Bird’s-Eye-View IoU to guide Non-Maximum Suppression (NMS), ensuring that proposals with high localization accuracy are prioritized over those with merely high classification scores.
  • Point-IoU-aware Refinement: This stage treats 3D IoU estimation as a point segmentation task. By checking how many grid points within a proposal fall inside the ground-truth box, the model achieves a more robust estimate of localization quality than simple regression.

Results and Discussion

The model was evaluated on the KITTI and ONCE benchmarks. On the KITTI test set, JPV-Net demonstrated significant gains over existing methods like PV-RCNN, particularly for smaller, complex objects. It achieved a 2.06% improvement in the "Cyclist" category and a 1.70% improvement in "Pedestrian" (Moderate difficulty), highlighting the importance of raw geometric detail.

Qualitative comparison showing JPV-Net's ability to eliminate false positives (red circles/rectangles) by leveraging joint point-voxel features.

Ablation studies confirmed that the fusion module alone contributed a 3.13% improvement over a voxel-only baseline. Qualitatively, JPV-Net excels at eliminating "ghost" detections—false positives where structures like walls or pillars are misidentified as vehicles or pedestrians—by utilizing the high-resolution point branch to correct voxel-level ambiguities.

Conclusion

JPV-Net successfully integrates voxel efficiency with point-based geometric precision. By treating points as a "correction" for lossy voxels and utilizing segmentation as a proxy for IoU, the model achieves state-of-the-art accuracy. While the dual-encoder approach increases computational complexity, the resulting gains in reliability are substantial for autonomous systems. Future work will focus on optimizing inference speeds to facilitate deployment on production-grade hardware.

Summary updated

Abstract

Voxel and point representations are widely applied in recent 3D object detection tasks from LiDAR point clouds. Voxel representations contribute to efficiently and rapidly locating objects, whereas point representations are capable of describing intra-object spatial relationship for detection refinement. In this work, we aim to exploit the strengths of both two representations, and present a novel two-stage detector, named Joint Point-Voxel Network (JPV-Net). Specifically, our framework is equipped with a Dual Encoders-Fusion Decoder, which consists of the dual encoders to extract voxel features of sketchy 3D scenes and point features rich in geometric context, respectively, and the Feature Propagation Fusion (FP-Fusion) decoder to attentively fuse them from coarse to fine. By making use of the advantages of these features, the refinement network can effectively eliminate false detection and provide better accuracy. Besides, to further develop the perception characteristics of voxel CNN and point backbone, we design two novel intersection-over-union (IoU) estimation modules for proposal generation and refinement, both of which can alleviate the misalignment between the localization and the classification confidence. Extensive experiments on the KITTI dataset and ONCE dataset demonstrate that our proposed JPV-Net outperforms other state-of-the-art methods with remarkable margins.
BibTeX
@inproceedings{song2022jpv,
  author = {Nan Song and Tianyuan Jiang and Jian Yao},
  title = {{JPV-Net: Joint Point-Voxel Representations for Accurate 3D Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=euvFfWxM20},
}
Figure