Voxelized 3D Feature Aggregation for Multiview Detection

Jiahao Ma, Jinguang Tong, Shan Wang, Wei Zhao, Liang Zheng, Chuong Nguyen

Abstract

Multi-view detection incorporates multiple camera views to alleviate occlusion in crowded scenes, where the state-of-the-art approaches adopt homography transformations to project multi-view features to the ground plane. However, we find that these 2D transformations do not take into account the object's height, and with this neglection features along the vertical direction of same object are likely not projected onto the same ground plane point, leading to impure ground-plane features. To solve this problem, we propose VFA, voxelized 3D feature aggregation, for feature transformation and aggregation in multi-view detection. Specifically, we voxelize the 3D space, project the voxels onto each camera view, and associate 2D features with these projected voxels. This allows us to identify and then aggregate 2D features along the same vertical line, alleviating projection distortions to a large extent. Additionally, because different kinds of objects (human vs. cattle) have different shapes on the ground plane, we introduce the oriented Gaussian encoding to match such shapes, leading to increased accuracy and efficiency. We perform experiments on multiview 2D detection and multiview 3D detection problems. Results on four datasets (including a newly introduced MultiviewC dataset) show that our system is very competitive compared with the state-of-the-art approaches.

Introduction

In crowded environments, single-camera detection often fails due to severe occlusion. Multiview detection addresses this by fusing information from multiple synchronized cameras. The standard approach for fusion involves using 2D homography to project image features onto a common ground plane.

However, this method suffers from a fundamental flaw: it assumes all objects are flat on the ground. Because real-world objects have height, perspective causes the top of an object to project to a different ground-plane coordinate than its base. This "projection distortion" results in smeared, impure feature maps that degrade detection performance and slow training convergence.

Comparison between 2D homography projection (top) and voxelized feature aggregation (bottom). Note how 2D homography smears features due to height, while VFA aggregates them vertically to a single point.

Methodology: Voxelized 3D Feature Aggregation (VFA)

To overcome projection distortion, we propose Voxelized 3D Feature Aggregation (VFA). Instead of a direct 2D-to-2D mapping, VFA introduces a 3D intermediate representation:

  1. Voxelization: The 3D space of interest is partitioned into a grid of cubic voxels.
  2. 3D-to-2D Projection: Each voxel is projected onto the 2D feature maps of all available camera views. Features are sampled from these projected locations.
  3. Vertical Aggregation: Features sampled for voxels sharing the same (x, y) ground coordinates but different z (height) values are aggregated. This ensures that all features belonging to the same vertical column in the real world are mapped to the same location on the final Bird’s-Eye-View (BEV) map.
  4. Oriented Gaussian Distribution (OGD): To better represent non-circular objects like cattle, we introduce OGD. This encoding uses oriented ellipses to represent object footprints, providing a more precise supervisory signal than standard point-based or circular representations.

Experimental Evaluation

We evaluated VFA across four benchmarks: Wildtrack and MultiviewX (pedestrian detection), MVM3D (mobile robot detection), and a new synthetic dataset, MultiviewC. MultiviewC was generated in Unreal Engine 4 and features 15 cattle in various poses across 7 camera views, providing a rigorous test for orientation and varied object sizes.

Key Results

VFA achieves state-of-the-art performance across all tested scenarios. On the Wildtrack dataset, VFA improved Multi-Object Detection Precision (MODP) by 12.5% over previous methods. On the new MultiviewC dataset, the combination of VFA and OGD achieved a 96.5% AP3D.

Crucially, the use of OGD led to a 4x speedup in training convergence. By aligning the ground-truth representation with the physical shape of the objects, the network learns the spatial relationships much more efficiently.

Conclusion

VFA addresses the long-standing issue of projection distortion in multiview detection by incorporating height-aware 3D voxel aggregation. By collapsing the vertical dimension after aggregation, the system maintains the efficiency of 2D detection heads while benefiting from 3D geometric consistency. Future work will focus on optimizing the memory-accuracy trade-off for real-time deployment and exploring the application of VFA to objects not grounded on a plane.

Summary updated

Abstract

Multi-view detection incorporates multiple camera views to alleviate occlusion in crowded scenes, where the state-of-the-art approaches adopt homography transformations to project multi-view features to the ground plane. However, we find that these 2D transformations do not take into account the object's height, and with this neglection features along the vertical direction of same object are likely not projected onto the same ground plane point, leading to impure ground-plane features. To solve this problem, we propose VFA, voxelized 3D feature aggregation, for feature transformation and aggregation in multi-view detection. Specifically, we voxelize the 3D space, project the voxels onto each camera view, and associate 2D features with these projected voxels. This allows us to identify and then aggregate 2D features along the same vertical line, alleviating projection distortions to a large extent. Additionally, because different kinds of objects (human vs. cattle) have different shapes on the ground plane, we introduce the oriented Gaussian encoding to match such shapes, leading to increased accuracy and efficiency. We perform experiments on multiview 2D detection and multiview 3D detection problems. Results on four datasets (including a newly introduced MultiviewC dataset) show that our system is very competitive compared with the state-of-the-art approaches. %Our code and data will be open-sourced.Code and MultiviewC are released at https://github.com/Robert-Mar/VFA.
BibTeX
@misc{ma2021voxelized,
  author = {Jiahao Ma and Jinguang Tong and Shan Wang and Wei Zhao and Liang Zheng and Chuong Nguyen},
  title = {{Voxelized 3D Feature Aggregation for Multiview Detection}},
  year = {2021},
  eprint = {2112.03471},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.03471},
}
Figure