HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects

Wei Xie, Zhipeng Yu, Zimeng Zhao, Binghui Zuo, Yangang Wang

Abstract

We present HMDO (Hand Manipulation with Deformable Objects), the first markerless multi-view dataset capturing interactive motions between hands and deformable objects. Reconstructing such interactions is challenging due to mutual occlusion, entangled visual features, and complex non-rigid deformations. To address these, we propose a collaborative annotation method where estimated hand features guide global object pose estimation, followed by an optimization process that analyzes hand-object relationships to reconstruct deformations. HMDO contains 21,600 frames across 12 sequences, providing a high-quality benchmark for learning-based reconstruction of deformable interaction scenes.

Introduction

While 3D reconstruction of hands interacting with rigid objects has seen significant progress, capturing interactions with deformable objects remains a major hurdle in computer vision. Real-world tasks frequently involve "squishy" objects—squeezing a sponge, handling food, or interacting with soft toys. Existing datasets primarily focus on rigid bodies, leaving a gap in training models for non-rigid scenarios. The core difficulties lie in the fact that hands often occlude the very regions undergoing deformation, and the infinite degrees of freedom in soft bodies make traditional tracking methods insufficient.

The HMDO Dataset

The HMDO dataset was captured using a custom multi-view system consisting of 10 synchronized high-speed industrial cameras. The dataset features 12 distinct "zodiac" plush dolls, chosen for their varied and complex morphologies (e.g., ears, tails, limbs).

  • Scale: 21,600 frames across 12 sequences.
  • Variety: Includes multiple perspectives, diverse object shapes, and various interaction types like poking, squeezing, and grasping.
  • Annotations: Provides 3D meshes for both the hand and the object, along with contact deformation maps.

Collaborative Reconstruction Pipeline

To accurately annotate the captured data, we developed a top-down collaborative reconstruction framework. The pipeline avoids physical markers, ensuring the visual data remains representative of real-world conditions.

Overview of the HMDO dataset and the collaborative reconstruction pipeline.

The process begins with Hand Tracking, utilizing a 2D pose estimation network (SRNet) to identify keypoints, which are then lifted into 3D to fit a high-resolution hand mesh. Simultaneously, an encoder-decoder network performs Object Segmentation to isolate the target from the background.

The "collaborative" aspect comes into play during Global Pose Optimization and Deformation Optimization. By treating the hand as a physical constraint, the system can infer deformations in occluded areas. If the hand mesh penetrates the object's initial volume, the algorithm pushes the object's vertices out of the hand's path. This local deformation is then propagated across the object's surface using Embedded Deformation Graphs and geodesic distances, ensuring a smooth and realistic result.

Results and Discussion

Evaluation of the HMDO pipeline demonstrates high-fidelity reconstruction. The hand tracking achieves an average joint error of less than 6mm. For object tracking, the system maintains a mean Intersection over Union (mIoU) of approximately 87% for global pose estimation. Qualitatively, the method successfully captures complex deformations on irregular surfaces, such as the ears of a rabbit or the snout of a pig doll.

However, limitations remain. The current model assumes uniform material properties, which may not hold for objects with varying internal stiffness. Additionally, extreme deformations—such as significant twisting or folding—remain a challenge for the current optimization framework.

Conclusion

HMDO fills a critical void in the field of hand-object interaction research. By providing a high-quality, markerless dataset and a robust reconstruction pipeline, this work paves the way for more advanced AI models capable of understanding and interacting with the soft, deformable world around us. Future work will explore incorporating depth data and multi-material modeling to further enhance reconstruction accuracy.

Summary updated

Abstract

We construct the first markerless deformable interaction dataset recording interactive motions of the hands and deformable objects, called HMDO (Hand Manipulation with Deformable Objects). With our built multi-view capture system, it captures the deformable interactions with multiple perspectives, various object shapes, and diverse interactive forms. Our motivation is the current lack of hand and deformable object interaction datasets, as 3D hand and deformable object reconstruction is challenging. Mainly due to mutual occlusion, the interaction area is difficult to observe, the visual features between the hand and the object are entangled, and the reconstruction of the interaction area deformation is difficult. To tackle this challenge, we propose a method to annotate our captured data. Our key idea is to collaborate with estimated hand features to guide the object global pose estimation, and then optimize the deformation process of the object by analyzing the relationship between the hand and the object. Through comprehensive evaluation, the proposed method can reconstruct interactive motions of hands and deformable objects with high quality. HMDO currently consists of 21600 frames over 12 sequences. In the future, this dataset could boost the research of learning-based reconstruction of deformable interaction scenes.
BibTeX
@misc{xie2023hmdo,
  author = {Wei Xie and Zhipeng Yu and Zimeng Zhao and Binghui Zuo and Yangang Wang},
  title = {{HMDO: Markerless Multi-view Hand Manipulation Capture with Deformable Objects}},
  year = {2023},
  eprint = {2301.07652},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.07652},
}
Figure