Introduction
Reconstructing high-quality 3D models of objects being manipulated by human hands is a long-standing challenge in computer vision, with significant implications for AR/VR and robotics. Traditional techniques like Structure-from-Motion (SfM) typically rely on static scenes and rich surface textures. However, when an object is held, the scene becomes dynamic, and the object is frequently occluded by the hand. Furthermore, many everyday items lack the distinct visual patterns required by SfM to "anchor" points in 3D space. This research introduces a novel pipeline designed to overcome these hurdles using monocular RGB video.
Methodology
The proposed framework moves away from raw pixel matching, instead leveraging semantic understanding and generative priors to reconstruct objects.
- Generative 3D Priors: The system utilizes a Text-to-3D model to generate a "rough draft" of the object based on a simple text description (e.g., "a yellow rubber duck").
- Semantic Initialization: To align this draft with the video, the authors use DINO features. Unlike standard pixels, these features capture high-level semantic meaning, allowing the system to estimate the object's initial pose even if the generated prior's texture or exact dimensions do not perfectly match the real-world object.
- Joint Optimization: The object's shape and its movement (pose) are refined simultaneously using a Neural Implicit Representation (specifically a Signed Distance Function or SDF), which enables the creation of smooth, high-detail surfaces.
- Pose Outlier Voting: To handle heavy hand occlusions, the system employs a "Pose Outlier Voting" strategy. By using geometric error metrics (Sampson distance) and rendering quality (PSNR), the system detects "bad" frames where tracking has failed and resets them using information from neighboring successful frames.
Key Results and Evaluation
The method was evaluated on three major real-world datasets: HO3D, HOD, and HOPE (which features challenging two-hand interactions). The proposed pipeline significantly outperformed state-of-the-art baselines like HOLD and IHOI:
- Accuracy: On the HO3D dataset, the method achieved a Chamfer Distance (CD) of 0.51 $cm^2$, compared to 1.82 $cm^2$ for the previous best method.
- Pose Precision: Rotation errors (RPEr) were reduced by nearly 50% compared to existing baselines.
- Texture-less Success: The method succeeded on difficult, texture-less objects like scissors and rubber ducks where traditional methods typically fail.
- Complete Reconstruction: Because the object is rotated in the hand during the video, the system can reconstruct the entire object, including the base, which is usually impossible when scanning objects sitting on a flat surface.
Limitations and Future Work
While powerful, the method's success is partially tethered to the quality of the initial generative 3D prior. If the text-to-3D model produces a vastly different topology, the initialization phase may struggle. Additionally, perfectly symmetrical objects can create semantic ambiguity, potentially leading to "flipped" poses. Future research will explore integrating more advanced semantic correspondence techniques to further improve robustness in unposed, "in-the-wild" scenarios.
Summary updated