Unsupervised Learning of Hierarchical Models for Hand-Object Interactions

Xu Xie, Hangxin Liu, Mark Edmonds, Feng Gao, Siyuan Qi, Yixin Zhu, Brandon Rothrock, Song-Chun Zhu

Abstract

Contact forces of the hand are visually unobservable, but play a crucial role in understanding hand-object interactions. In this paper, we propose an unsupervised learning approach for manipulation event segmentation and manipulation event parsing. The proposed framework incorporates hand pose kinematics and contact forces using a low-cost easy-to-replicate tactile glove. We use a temporal grammar model to capture the hierarchical structure of events, integrating extracted force vectors from the raw sensory input of poses and forces. The temporal grammar is represented as a temporal And-Or graph (T-AOG), which can be induced in an unsupervised manner. We obtain the event labeling sequences by measuring the similarity between segments using the Dynamic Time Alignment Kernel (DTAK). Experimental results show that our method achieves high accuracy in manipulation event segmentation, recognition and parsing by utilizing both pose and force data.
BibTeX
@inproceedings{xie2018unsupervised,
  author = {Xu Xie and Hangxin Liu and Mark Edmonds and Feng Gao and Siyuan Qi and Yixin Zhu and Brandon Rothrock and Song-Chun Zhu},
  title = {{Unsupervised Learning of Hierarchical Models for Hand-Object Interactions}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  doi = {10.1109/icra.2018.8461214},
}

Loading related content...

Figure