ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

Danila Rukhovich, Anna Vorontsova, Anton Konushin

Overview

ImVoxelNet is a unified, fully convolutional framework designed for 3D object detection from posed RGB images. By projecting 2D image features into a shared 3D voxel volume, it creates a "general-purpose" architecture capable of state-of-the-art performance across both indoor and outdoor datasets. It handles both monocular and multi-view inputs within a single end-to-end pipeline.

Introduction

Detecting 3D objects from standard RGB images is an "ill-posed" problem due to the lack of explicit depth and scale information in 2D pixels. Traditionally, research has been bifurcated:

  1. Outdoor Detection: Often focuses on cars on a flat ground plane, typically simplifying the problem by projecting data onto a 2D "Bird’s Eye View" (BEV).
  2. Indoor Detection: Deals with objects at varying heights and orientations, requiring more complex 3D spatial reasoning.

Furthermore, while multi-view data should enhance 3D detection, most existing methods rely on late fusion of independent 2D predictions. ImVoxelNet addresses these challenges by building a true 3D representation early in the pipeline, allowing it to generalize across environments and scale seamlessly from single to multiple camera views.

Architecture and Methodology

The core innovation of ImVoxelNet is its Image-to-Voxel projection pipeline.

The ImVoxelNet architecture: 2D features are extracted, unprojected into a 3D volume, processed by a 3D CNN, and passed to domain-specific heads.
  1. 2D Feature Extraction: Each input image is processed by a standard 2D backbone (ResNet-50 with a Feature Pyramid Network) to extract multi-scale visual features.
  2. Voxel Unprojection: Using known camera intrinsics and extrinsics, 2D features are "unprojected" into a 3D voxel grid. For multi-view inputs, features from different images falling into the same voxel are averaged, creating a unified 3D feature volume.
  3. 3D Convolutional Neck: A 3D CNN encoder-decoder refines these spatial features, helping the model learn geometry and interpolate features in occluded or "invisible" areas.
  4. Domain-Specific Heads: To maintain generality, the architecture uses two specialized heads:
  5. Outdoor Head: Collapses the 3D volume into a 2D BEV map and uses an anchor-based detection head optimized for ground-plane objects.
  6. Indoor Head: Employs a dense, multi-scale 3D head (inspired by FCOS) to detect objects anywhere in 3D space.

Key Insights

  • Voxels as a Universal Language: Converting 2D images into a 3D voxel grid early allows the model to leverage "neck" and "head" logic developed for LiDAR-based detectors, bridging the gap between RGB and point-cloud research.
  • End-to-End Multi-View Training: Unlike late-fusion methods, ImVoxelNet optimizes the 3D representation directly from multiple images. Performance scales significantly with more views; for instance, jumping from 9.1 mAP with 1 image to 46.6 mAP with 50 images on ScanNet.
  • Geometric Hallucination: The 3D CNN neck is crucial for learning to interpolate geometry in regions occluded in specific camera views.

Experimental Results

The authors evaluated ImVoxelNet on four major benchmarks: KITTI (monocular outdoor), nuScenes (multi-view outdoor), SUN RGB-D (monocular indoor), and ScanNet (multi-view indoor).

Qualitative results of ImVoxelNet on outdoor datasets, showing accurate 3D bounding box predictions for vehicles.
  • Outdoor (KITTI): Achieved 17.15% AP (Moderate) for cars, outperforming contemporary RGB-only monocular methods.
  • Outdoor (nuScenes): In multi-view settings, it surpassed previous state-of-the-art (MonoDIS) by over 1% mAP, showing nearly double the accuracy at strict distance thresholds ([email protected]).
  • Indoor (SUN RGB-D & ScanNet): On SUN RGB-D, it reached 43.66 mAP, a significant improvement over previous monocular methods. On ScanNet, it established a new benchmark for multi-view RGB detection at 48.1 mAP.

Limitations and Future Work

The primary trade-off for ImVoxelNet’s accuracy is its computational cost. Processing 3D voxel volumes with 3D convolutions is memory-intensive and slower than 2D-only methods, with the 3D neck taking between 0.14s and 2.45s per scene.

Additionally, the method requires posed images (accurate camera calibration and localization). Future research could integrate modules to estimate these poses directly from images, enhancing robustness for "in-the-wild" applications where camera parameters are unknown.

Summary updated

Abstract

In this paper, we introduce the task of multi-view RGB-based 3D object detection as an end-to-end optimization problem. To address this problem, we propose ImVoxel-Net, a novel fully convolutional method of 3D object detection based on posed monocular or multi-view RGB images. The number of monocular images in each multi-view input can variate during training and inference; actually, this number might be unique for each multi-view input. ImVoxelNet successfully handles both indoor and outdoor scenes, which makes it general-purpose. Specifically, it achieves state-of-the-art results in car detection on KITTI (monocular) and nuScenes (multi-view) benchmarks among all methods that accept RGB images. Moreover, it surpasses existing RGB-based 3D object detection methods on the SUN RGB-D dataset. On ScanNet, ImVoxelNet sets a new benchmark for multi-view 3D object detection. The source code and the trained models are available at https://github.com/saic-vul/imvoxelnet.
BibTeX
@inproceedings{rukhovich2022imvoxelnet,
  author = {Danila Rukhovich and Anna Vorontsova and Anton Konushin},
  title = {{ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00133},
}
Figure