MEgATrack
Lists
Only you see this
Abstract
We present a system for real-time hand-tracking to drive virtual and augmented reality (VR/AR) experiences. Using four fisheye monochrome cameras, our system generates accurate and low-jitter 3D hand motion across a large working volume for a diverse set of users. We achieve this by proposing neural network architectures for detecting hands and estimating hand keypoint locations. Our hand detection network robustly handles a variety of real world environments. The keypoint estimation network leverages tracking history to produce spatially and temporally consistent poses. We design scalable, semi-automated mechanisms to collect a large and diverse set of ground truth data using a combination of manual annotation and automated tracking. Additionally, we introduce a detection-by-tracking method that increases smoothness while reducing the computational cost; the optimized system runs at 60Hz on PC and 30Hz on a mobile processor. Together, these contributions yield a practical system for capturing a user's hands and is the default feature on the Oculus Quest VR headset powering input and social presence.
BibTeX
@article{han2020megatrack,
author = {Shangchen Han and Beibei Liu and Randi Cabezas and Christopher D. Twigg and Peizhao Zhang and Jeff Petkau and Tsz-Ho Yu and Chun-Jung Tai and Muzaffer Akbay and Zheng Wang and Asaf Nitzan and Gang Dong and Yuting Ye and Lingling Tao and Chengde Wan and Robert Wang},
title = {{MEgATrack}},
journal = {ACM Transactions on Graphics},
year = {2020},
doi = {10.1145/3386569.3392452},
}Loading related content...