Abstract
This paper introduces a novel framework for high-fidelity 3D reconstruction of objects held in a moving hand using only a single, static RGB camera. By leveraging hand-tracking as a proxy for camera motion and employing implicit neural representations (SDFs), the authors turn a simple monocular video into a detailed 3D scanner without requiring any prior knowledge of the object's category or shape.
Introduction
Reconstructing 3D objects from 2D images is a cornerstone of computer vision, but the "hand-held" scenario presents unique challenges. Traditional Structure from Motion (SfM) algorithms assume a static scene; when an object moves independently of the background, SfM fails. Conversely, single-view reconstruction methods often rely on learned priors, limiting their effectiveness to known object categories.
Hand-held objects are often small, textureless, and significantly occluded by fingers, making direct 6DoF pose tracking nearly impossible. This research aims to bridge this gap, enabling high-quality 3D modeling of arbitrary objects simply by rotating them in front of a standard camera.
Methodology
The proposed two-stage pipeline treats the hand not as an obstacle, but as a reliable motion guide.
1. Hand Tracking
Instead of tracking the object directly, the system tracks the hand using the MANO parametric model. To ensure robustness against occlusion, hand parameters (pose and shape) are optimized across the entire video sequence simultaneously. This global optimization provides the relative "camera" poses necessary for multi-view reconstruction.
2. Dense Reconstruction
The system utilizes an implicit neural representation based on NeuS, representing the 3D scene as a Signed Distance Field (SDF). Three key modules enhance this process for hand-held scenarios:
- Pose Refinement: Jointly optimizes camera poses and geometry to "de-jitter" the reconstruction.
- Deformation Field: A learnable field compensates for non-rigid "wiggle" or tiny relative movements between the hand and object.
- Semantic-Guided Sampling: Uses a semantic head to identify the object, adaptively sampling more rays on the object itself to preserve fine details.
The Hand-held Object Dataset (HOD)
To evaluate the framework, the authors introduced the Hand-held Object Dataset (HOD), featuring 35 objects:
- Sculptures: Five complex items with high geometric detail and white textures.
- Daily Objects: 30 common items.
- Ground Truth: 14 objects were professionally scanned to provide high-fidelity evaluation data.
Unlike existing datasets like HO3D or ContactPose, HOD utilizes a monocular setup with realistic, daily-life objects rather than 3D-printed replicas.
Results and Key Insights
The core innovation lies in using the physical constraint between the hand and the object as a feature. By tracking the hand, the object's motion is effectively solved. Furthermore, global optimization across the video sequence dramatically increases robustness against heavy finger occlusions.
Quantitative and Qualitative Performance
The proposed method significantly outperforms both single-view learning methods and standard multi-view baselines. On the HOD dataset, it achieved a Chamfer Distance (CD) of 0.249, compared to 3.391 for the vanilla NeuS baseline and 6.802 for state-of-the-art single-view methods (IHOI). Qualitatively, the system recovers sharp edges and intricate details on objects like small robot toys and classical busts, where other methods often produce undifferentiated "blobs."
Limitations and Future Work
Despite its success, several challenges remain for consumer-level deployment:
- Computation Time: Neural SDF reconstruction currently requires approximately 30 hours for a one-minute video. Future work may explore faster techniques like Instant-NGP.
- Post-Processing: The system relies on Poisson Reconstruction to fill holes left by the hand's grip, which can struggle with very thin or complex structures.
- Rigid Grasp Assumption: The model assumes a fixed grasping gesture. Handling "re-grasping" or shifting grips mid-video is a key direction for future research.
Summary updated