Reconstruct, Rasterize and Backprop: Dense shape and pose estimation from a single image

Aniket Pokale, Aditya Aggarwal, Krishna Murthy Jatavallabhula, K. Madhava Krishna

Abstract: This paper presents a new system to obtain dense object reconstructions along with 6-DoF poses from a single image. Geared towards high fidelity reconstruction, several recent approaches leverage implicit surface representations and deep neural networks to estimate a 3D mesh of an object, given a single image. However, all such approaches recover only the shape of an object; the reconstruction is often in a canonical frame, unsuitable for downstream robotics tasks. To this end, we leverage recent advances in differentiable rendering (in particular, rasterization) to close the loop with 3D reconstruction in camera frame. We demonstrate that our approach---dubbed reconstruct, rasterize and backprop (RRB) achieves significantly lower pose estimation errors compared to prior art, and is able to recover dense object shapes and poses from imagery. We further extend our results to an (offline) setup, where we demonstrate a dense monocular object-centric egomotion estimation system.

Introduction

Recovering the 3D geometry and 6-DoF pose of objects from monocular imagery is a fundamental problem in computer vision with significant implications for robotics. While recent advances in deep learning have enabled high-fidelity 3D reconstructions using implicit surface representations, these methods typically operate in a "canonical" coordinate frame. This abstraction limits their utility for real-world interaction, where the object's pose relative to the camera is critical. Conversely, traditional pose estimation methods often rely on simplified geometric proxies like bounding boxes, sacrificing fine-grained detail.

The "Reconstruct, Rasterize and Backprop" (RRB) framework addresses this disconnect by integrating high-fidelity reconstruction with a differentiable rendering loop. By optimizing the object's pose in the camera frame to match observed image features, RRB recovers both dense geometry and precise spatial orientation.

The RRB Framework

The RRB pipeline closes the loop between 3D reconstruction and 2D observation through a differentiable rendering process. It transitions from a single RGB image to a refined 3D mesh in camera coordinates via four main stages:

The RRB framework overview: A neural network predicts initial shape and pose, which are then refined through a differentiable rasterization engine that minimizes the silhouette loss between the rendered model and the input image.
  1. Occupancy Network: The system uses a deep neural network to predict the 3D shape of the object using an occupancy representation, which determines if a point in space is inside or outside the object, eventually producing a high-fidelity triangle mesh.
  2. Viewpoint Initialization: A secondary network (based on RenderForCNN) provides an initial "rough guess" of the camera's azimuth and elevation relative to the object.
  3. Differentiable Rasterization: This is the core of the RRB approach. The system takes the predicted mesh and the initial pose guess to render a 2D silhouette. Because the rendering process is differentiable, the system can compute gradients based on the discrepancy between the rendered silhouette and the actual object mask in the input image.
  4. Optimization via Backpropagation: These gradients are used to iteratively refine the pose parameters (rotation and translation). By "nudging" the pose to minimize the silhouette loss, the system aligns the 3D reconstruction with the 2D observation in the camera frame.

Key Insights

  • Analysis-by-Synthesis: RRB leverages the principle that a correct 3D pose should accurately reproduce the 2D observation when rendered.
  • Hybrid Optimization: The framework combines the global search capabilities of neural networks for initialization with the local precision of classical optimization via differentiable rendering.
  • Coordinate Frame Alignment: By incorporating camera height and bounding box constraints, the system successfully transforms reconstructions from an abstract canonical space into the practical camera coordinate system.

Evaluation and Results

The authors validated RRB using the ShapeNet dataset, rendered against complex backgrounds from SUN397, and custom synthetic Blender sequences for egomotion tasks. The RRB method demonstrated significant improvements over baseline initialization:

  • Pose Accuracy: Azimuth error was reduced from 61.17° (initial guess) to 10.79°.
  • Spatial Recovery: The system achieved a mean translation error of 0.63 meters.
  • Reconstruction Quality: The 3D Intersection over Union (IoU) reached 0.7795, confirming high-fidelity shape recovery.
  • Egomotion Estimation: In tracking scenarios, RRB outperformed the industry-standard ORB-SLAM in terms of Absolute Trajectory Error (ATE) by utilizing dense object-level features.

Limitations and Future Work

Despite its performance, the RRB framework faces challenges in specific scenarios, particularly regarding geometric complexity and environmental assumptions.

Reconstruction results across various chair models. The bottom row illustrates failure cases where thin or disconnected structures (e.g., chair legs) are not perfectly captured by the occupancy network, impacting the rendering loop.
  • Thin/Disconnected Structures: Slender structures, such as chair legs, can be difficult for occupancy networks to represent accurately, potentially leading to errors in the rendering-based optimization.
  • Scale Ambiguity: The system currently requires external constraints, such as known camera height, to resolve the inherent ambiguity between an object's size and its distance from the camera.
  • Segmentation Dependency: The optimization loop relies on accurate object silhouettes. Future iterations could integrate automated instance segmentation to enable robust performance on uncurated, real-world imagery.

Summary updated

Abstract

This paper presents a new system to obtain dense object reconstructions along with 6-DoF poses from a single image. Geared towards high fidelity reconstruction, several recent approaches leverage implicit surface representations and deep neural networks to estimate a 3D mesh of an object, given a single image. However, all such approaches recover only the shape of an object; the reconstruction is often in a canonical frame, unsuitable for downstream robotics tasks. To this end, we leverage recent advances in differentiable rendering (in particular, rasterization) to close the loop with 3D reconstruction in camera frame. We demonstrate that our approach---dubbed reconstruct, rasterize and backprop (RRB) achieves significantly lower pose estimation errors compared to prior art, and is able to recover dense object shapes and poses from imagery. We further extend our results to an (offline) setup, where we demonstrate a dense monocular object-centric egomotion estimation system.
BibTeX
@inproceedings{pokale2020reconstruct,
  author = {Aniket Pokale and Aditya Aggarwal and Krishna Murthy Jatavallabhula and K. Madhava Krishna},
  title = {{Reconstruct, Rasterize and Backprop: Dense shape and pose estimation from a single image}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=moikAZcXU9},
}
Figure