Deep Cuboid Detection: Beyond 2D Bounding Boxes

Debidatta Dwibedi, Tomasz Malisiewicz, Vijay Badrinarayanan, Andrew RabinovicharXiv 2016 · arXiv

Abstract

Standard object detection frameworks typically output 2D bounding boxes, which lack the 3D orientation and geometric depth required for advanced robotics and augmented reality applications. This paper introduces an end-to-end deep learning system designed to detect 3D cuboids in single RGB images. By extending the Faster R-CNN architecture to regress eight vertices and employing a novel iterative feature pooling technique, the proposed method achieves state-of-the-art results in class-agnostic 3D localization.

Introduction

While 2D bounding boxes are a staple of computer vision, they provide limited information about an object's spatial configuration. Traditional methods for 3D reconstruction from 2D images often rely on low-level cues like vanishing points or edges, which frequently fail in cluttered real-world environments. This research bridges the gap by treating the "cuboid" as a fundamental geometric primitive. By training a deep network to recognize "cuboid-ness," the system can localize objects like appliances, furniture, and shipping containers regardless of their specific category.

Methodology

The Deep Cuboid Detector is built upon the Faster R-CNN framework, incorporating several key innovations to handle 3D geometry.

Architecture and Multi-Task Regression

The system utilizes a convolutional backbone (e.g., VGG16) for feature extraction. A Region Proposal Network (RPN) identifies candidate regions. Unlike standard detectors that predict four parameters for a 2D box, this model performs multi-task regression to predict 16 values, representing the $(x, y)$ coordinates for all eight vertices of the cuboid.

Iterative Feature Pooling

A core contribution of this work is the iterative refinement of detections. After an initial pass, the network generates a rough estimate of the cuboid's location. The system then "re-pools" features from this refined region and executes the regression again. This iterative process allows the network to focus its attention on the object's boundaries, significantly improving the precision of corner localization.

The iterative feature pooling architecture refines cuboid detections through multiple passes.

Experimental Results

The model was evaluated on the SUN Primitive dataset, which contains over 3,500 images of indoor and outdoor scenes with ground-truth 3D annotations.

Performance and Accuracy

The VGG16-based model with iterative pooling achieved a Mean Average Precision (mAP) of 75.47, representing a substantial improvement over classical HOG-based methods (approx. 24.0 mAP). The iterative refinement alone contributed a 7% increase in the Probability of Correct Keypoint (PCK).

The detector successfully localizes 3D cuboids for various objects in a cluttered office environment.

Efficiency

The system is designed for practical deployment. Using a lighter VGG-M backbone, the detector achieves a processing speed of 14 frames per second (fps) on a Titan Z GPU, making it viable for near real-time applications in robotics and AR.

Discussion and Conclusion

The findings suggest that deep convolutional neural networks can implicitly learn complex geometric concepts such as perspective and vanishing points directly from data. While the model occasionally faces challenges with viewpoint ambiguity (e.g., distinguishing identical faces of a cube), the over-parameterized approach of predicting all eight vertices allows the network to infer occluded corners based on visible visual evidence. Future work will focus on mobile optimization and the use of synthetic datasets to further enhance generalization.

Summary updated

Abstract

We present a Deep Cuboid Detector which takes a consumer-quality RGB image of a cluttered scene and localizes all 3D cuboids (box-like objects). Contrary to classical approaches which fit a 3D model from low-level cues like corners, edges, and vanishing points, we propose an end-to-end deep learning system to detect cuboids across many semantic categories (e.g., ovens, shipping boxes, and furniture). We localize cuboids with a 2D bounding box, and simultaneously localize the cuboid's corners, effectively producing a 3D interpretation of box-like objects. We refine keypoints by pooling convolutional features iteratively, improving the baseline method significantly. Our deep learning cuboid detector is trained in an end-to-end fashion and is suitable for real-time applications in augmented reality (AR) and robotics.
BibTeX
@misc{dwibedi2016deep,
  author = {Debidatta Dwibedi and Tomasz Malisiewicz and Vijay Badrinarayanan and Andrew Rabinovich},
  title = {{Deep Cuboid Detection: Beyond 2D Bounding Boxes}},
  year = {2016},
  eprint = {1611.10010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.10010},
}
Figure