Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera

Jiye Lee, Hanbyul Joo

Introduction

Traditional high-fidelity motion capture (mocap) has long been confined to laboratory settings, requiring expensive optical arrays or cumbersome suits with 17 or more inertial measurement unit (IMU) sensors. While recent research has attempted to reduce this hardware burden to six sensors, these setups remain impractical for daily use and often struggle with accuracy in real-world environments.

Researchers have now introduced a breakthrough system called "Mocap Everyone Everywhere," which achieves high-quality 3D full-body tracking using only three consumer-grade devices: two smartwatches and a single head-mounted action camera. By fusing sparse inertial data with visual SLAM (Simultaneous Localization and Mapping), this system enables robust motion capture in complex "in-the-wild" scenarios like stairs and hills.

Overview of the lightweight hardware setup and its ability to reconstruct motion in diverse outdoor environments.

Overcoming the Challenges of Sparse Mocap

The primary hurdle for sparse sensor setups (e.g., just the head and wrists) is ambiguity. With limited data points, it is difficult for a system to distinguish between similar upper-body movements that might correspond to different lower-body actions, such as sitting versus standing. Furthermore, IMU-based systems are notorious for root drift, where small, accumulating sensor errors cause the estimated global position of the subject to "float" away from their actual location over time.

The Proposed Framework

The "Mocap Everyone Everywhere" system addresses these issues through three innovative pillars:

  1. SLAM-based Head Anchoring: Instead of relying on error-prone IMUs for global positioning, the system uses monocular SLAM (specifically DROID-SLAM) on the egocentric video feed. This provides a stable 6-degree-of-freedom (6DoF) trajectory of the head, serving as a global anchor that effectively eliminates root drift.
  2. Dynamic Floor-Level Updating: Traditional mocap models often assume a perfectly flat floor. To handle real-world terrain, the authors developed an algorithm that detects foot-to-ground contact and projects those points onto the SLAM-generated 3D point cloud. This allows the system to accurately track vertical movement on stairs or uneven ground.
  3. Multi-stage Transformer Regression: The motion estimation module utilizes a Transformer-based architecture. It first predicts the positions of "end-effectors" (hands and feet) as an intermediate step before regressing the full skeletal pose, helping the model better interpret the relationship between sparse inputs and complex body movements.

Social Mocap and Visual Optimization

A unique feature of this system is its ability to leverage "social" cues. In multi-person scenarios, a nearby user's camera can act as a third-person sensor, providing additional 2D visual cues to refine another person's pose. This turns a group of users into a distributed sensor network, ensuring movements remain natural and physically plausible even when individual sensors are occluded.

The system supports multi-person tracking and complex interactions, such as sitting, in unstructured environments.

Key Results and Performance

The system demonstrates a significant leap over existing state-of-the-art (SOTA) methods, even those using more sensors:

  • Root Accuracy: On the TotalCapture dataset, the system reduced Root Position Error by over 90% compared to 6-IMU baselines, dropping from approximately 34cm to just 3.25cm.
  • Pose Quality: The root-relative joint error (r.MPJPE) was recorded at 3.77cm, outperforming more complex sensor configurations.
  • Terrain Robustness: Qualitative evaluations show the system successfully tracking users navigating multiple flights of stairs and interacting with objects—scenarios where traditional "flat-floor" models typically fail.

Limitations and Future Work

While the results are promising, the system currently depends on the presence of visual features for SLAM to function; featureless environments or extreme motion blur can disrupt tracking. Additionally, the model currently uses a "mean" body shape, which may lead to minor artifacts for users with significantly different proportions. Future iterations aim to automate height scaling and improve robustness to varied body types.

In summary, this work represents a major step toward democratizing motion capture, moving it out of the lab and into the pockets (and onto the wrists) of everyday users.

Summary updated

Abstract

We present a lightweight and affordable motion capture method based on two smartwatches and a head-mounted camera. In contrast to the existing approaches that use six or more expert-level IMU devices, our approach is much more cost-effective and convenient. Our method can make wearable motion capture accessible to everyone everywhere, enabling 3D full-body motion capture in diverse environments. As a key idea to overcome the extreme sparsity and ambiguities of sensor inputs with different modalities, we integrate 6D head poses obtained from the head-mounted cameras for motion estimation. To enable capture in expansive indoor and outdoor scenes, we propose an algorithm to track and update floor level changes to define head poses, coupled with a multi-stage Transformer-based regression module. We also introduce novel strategies leveraging visual cues of egocentric images to further enhance the motion capture quality while reducing ambiguities. We demonstrate the performance of our method on various challenging scenarios, including complex outdoor environments and everyday motions including object interactions and social interactions among multiple individuals.
BibTeX
@inproceedings{lee2024mocap,
  author = {Jiye Lee and Hanbyul Joo},
  title = {{Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure