Abstract
Estimating the contact and deformation between human hands and nonrigid objects from a single RGB image is a complex task due to the inherent difficulty in aligning 3D geometry with 2D image features. This paper presents RUFormer (Regional Unwrapping Transformer), a novel framework that addresses this by "unwrapping" 3D surfaces into 2D regional profiles (RUPs) aligned with hand anatomy. This approach enables a Vision Transformer to exploit spatial correlations and predict realistic deformations. Evaluated on several benchmarks, RUFormer demonstrates superior performance in reducing mesh penetration and improving contact plausibility compared to state-of-the-art rigid-only methods.
Introduction
While significant progress has been made in reconstructing hand interactions with rigid objects, most everyday items—such as plush toys or plastic bottles—are nonrigid. The core challenge in modeling these interactions lies in representation. Existing methods often project contact data onto unordered 3D point clouds or meshes, making it difficult for neural networks to align 3D geometry with 2D image pixels. When an object deforms, the spatial consistency relied upon by standard learning models is broken.
Methods
The authors propose a two-stage solution involving a novel data representation and a specialized Transformer architecture.
Region Unwrapping Profiles (RUPs)
To bridge the gap between 3D meshes and 2D image cues, the hand and object surfaces are divided into 16 regions based on the kinematic bones of the hand (e.g., fingertips, joints, palm). These 3D sub-surfaces are then "unwrapped" into 2D images called Regional Unwrapping Profiles (RUPs).
- Hand-RUPs: Capture the hand surface.
- Object-RUPs: Record the nearest object surface intersections.
- Object-f-RUPs: Record the farthest object surface intersections to provide depth and deformation context.
By converting 3D data into 2D grids, the model can utilize standard image-based feature extractors and ensure that features are pixel-aligned across different regions.
RUFormer Architecture
The RUFormer architecture consists of two main components:
- Encoder (Contact Estimation): This stage takes image patches, bone transformations, and RUPs as input. A Vision Transformer (ViT) exploits correlations across the 16 hand regions to predict contact maps and a "deformed degree" (representing the level of compression in a region).
- Decoder (Deformation Estimation): This stage predicts fine-grained 3D transformations for sampling points on the object surface. It employs a deformation graph approach to ensure the entire object surface moves realistically based on predicted nodes.
Results and Discussion
RUFormer was evaluated on several key benchmarks, including HMDO (nonrigid), HO3D, FPHB (rigid), and ContactPose (contact accuracy).
Quantitative Performance
On the HMDO dataset, RUFormer significantly reduced the "Max Penetration" (the extent to which the hand mesh clips through the object) from a previous best of 12.78mm to 10.42mm. It also achieved the lowest intersection volume among compared methods. Even on rigid datasets like HO3D, RUFormer outperformed state-of-the-art rigid-only methods, achieving a Mean Per-Joint Position Error (MPJPE) of 14.78mm.
Qualitative Success
Visual results demonstrate that RUFormer produces much more physically plausible grasps. Unlike previous methods where fingers might disappear into an object mesh, RUFormer correctly predicts indentations on the object surface where the fingers exert pressure.
Conclusion
RUFormer introduces a robust way to handle nonrigid hand-object interactions by leveraging anatomy-driven regional unwrapping. By converting 3D contact problems into 2D image-aligned tasks, it allows Transformers to learn complex spatial relationships effectively. While the method currently relies on accurate initial 2D pose estimation and has higher computational costs, it sets a new standard for realistic contact and deformation estimation in monocular RGB images.
Summary updated