Abstract
Accurate 3D reconstruction of hands and held objects is essential for seamless human-robot interaction and immersive AR/VR experiences. However, current approaches often face a trade-off between the speed of single-view methods and the precision of dense multi-view optimization. This paper introduces SVHO (Sparse Multi-View Hand-Object reconstruction), a framework designed to reconstruct 3D meshes from a limited set of RGB images (2–8 views). By utilizing a learned codebook of 3D shape patches and a classification-based approach, SVHO generalizes to unseen objects in real-world environments without requiring per-view optimization.
Introduction
In applications like robotic grasping, a system must understand the precise geometry of both the human hand and the object being manipulated. Single-view methods frequently fail due to severe occlusions, while dense multi-view techniques like NeRF are computationally expensive and often require training on specific object instances. SVHO bridges this gap by focusing on the sparse multi-view setting, providing a fast, feed-forward solution that remains robust to novel object geometries.
Methodology: The SVHO Framework
The SVHO pipeline avoids the pitfalls of direct mesh regression by treating reconstruction as a classification problem over a discrete latent space.
Patchwise VQ-VAE and Codebooks
At the heart of the system is a Patchwise Vector Quantized-Variational Autoencoder (VQ-VAE). Instead of predicting a continuous surface, the model learns a "vocabulary" or codebook ($B_h$ for hands, $B_o$ for objects) of small 3D patches. The reconstruction task is simplified to identifying which codebook index best fits each cell in a 3D grid.
Multi-View Aggregation
For each 3D point in a canonical wrist-centered space, the model projects the point back onto all available 2D input images to extract local features. These features are processed by an image encoder to produce probability distributions over the codebook indices. By averaging these probabilities across all camera views, the model achieves a consensus reconstruction that naturally integrates information from multiple perspectives without iterative optimization. The final 3D mesh is generated using a decoder and the Marching Cubes algorithm.
Key Insights and Results
The researchers trained SVHO on the synthetic ObMan dataset and evaluated its performance on the real-world DexYCB dataset, which contains 20 unique, unseen objects.
The Multi-View Performance Divergence
A significant finding of the study is the differing impact of additional camera views on hand versus object reconstruction. While hand reconstruction quality (measured by F-score) improved steadily as more views were added, object reconstruction exhibited a surprising trend.
As shown in the experimental results, object reconstruction accuracy typically peaks at just two views. Beyond this point, the inclusion of more camera angles often introduces "noise" from real-world background clutter. Because the model was trained on synthetic data with clean backgrounds, it occasionally misinterprets background objects as part of the held item, leading to a decrease in precision as more views are integrated.
Conclusion and Future Work
SVHO demonstrates the power of patch-based classification for robust, zero-shot 3D reconstruction in sparse-view settings. While the model successfully generalizes to real-world objects like power drills and mustard bottles, its sensitivity to background clutter highlights a clear path for future research. Implementing automated hand-object segmentation to mask out background distractors could allow the model to fully leverage the benefits of high-count multi-view data.
Summary updated