Introduction
Traditional high-fidelity motion capture (mocap) has long been confined to laboratory settings, requiring expensive optical arrays or cumbersome suits with 17 or more inertial measurement unit (IMU) sensors. While recent research has attempted to reduce this hardware burden to six sensors, these setups remain impractical for daily use and often struggle with accuracy in real-world environments.
Researchers have now introduced a breakthrough system called "Mocap Everyone Everywhere," which achieves high-quality 3D full-body tracking using only three consumer-grade devices: two smartwatches and a single head-mounted action camera. By fusing sparse inertial data with visual SLAM (Simultaneous Localization and Mapping), this system enables robust motion capture in complex "in-the-wild" scenarios like stairs and hills.
Overcoming the Challenges of Sparse Mocap
The primary hurdle for sparse sensor setups (e.g., just the head and wrists) is ambiguity. With limited data points, it is difficult for a system to distinguish between similar upper-body movements that might correspond to different lower-body actions, such as sitting versus standing. Furthermore, IMU-based systems are notorious for root drift, where small, accumulating sensor errors cause the estimated global position of the subject to "float" away from their actual location over time.
The Proposed Framework
The "Mocap Everyone Everywhere" system addresses these issues through three innovative pillars:
- SLAM-based Head Anchoring: Instead of relying on error-prone IMUs for global positioning, the system uses monocular SLAM (specifically DROID-SLAM) on the egocentric video feed. This provides a stable 6-degree-of-freedom (6DoF) trajectory of the head, serving as a global anchor that effectively eliminates root drift.
- Dynamic Floor-Level Updating: Traditional mocap models often assume a perfectly flat floor. To handle real-world terrain, the authors developed an algorithm that detects foot-to-ground contact and projects those points onto the SLAM-generated 3D point cloud. This allows the system to accurately track vertical movement on stairs or uneven ground.
- Multi-stage Transformer Regression: The motion estimation module utilizes a Transformer-based architecture. It first predicts the positions of "end-effectors" (hands and feet) as an intermediate step before regressing the full skeletal pose, helping the model better interpret the relationship between sparse inputs and complex body movements.
Social Mocap and Visual Optimization
A unique feature of this system is its ability to leverage "social" cues. In multi-person scenarios, a nearby user's camera can act as a third-person sensor, providing additional 2D visual cues to refine another person's pose. This turns a group of users into a distributed sensor network, ensuring movements remain natural and physically plausible even when individual sensors are occluded.
Key Results and Performance
The system demonstrates a significant leap over existing state-of-the-art (SOTA) methods, even those using more sensors:
- Root Accuracy: On the TotalCapture dataset, the system reduced Root Position Error by over 90% compared to 6-IMU baselines, dropping from approximately 34cm to just 3.25cm.
- Pose Quality: The root-relative joint error (r.MPJPE) was recorded at 3.77cm, outperforming more complex sensor configurations.
- Terrain Robustness: Qualitative evaluations show the system successfully tracking users navigating multiple flights of stairs and interacting with objects—scenarios where traditional "flat-floor" models typically fail.
Limitations and Future Work
While the results are promising, the system currently depends on the presence of visual features for SLAM to function; featureless environments or extreme motion blur can disrupt tracking. Additionally, the model currently uses a "mean" body shape, which may lead to minor artifacts for users with significantly different proportions. Future iterations aim to automate height scaling and improve robustness to varied body types.
In summary, this work represents a major step toward democratizing motion capture, moving it out of the lab and into the pockets (and onto the wrists) of everyday users.
Summary updated