RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture

Xiaoyu Pan, Bowen Zheng, Xinwei Jiang, Zijiao Zeng, Qilong Kou, He Wang, Xiaogang Jin

Abstract

Optical motion capture (MoCap) is the "gold standard" for accurately capturing full-body motions. To make use of raw MoCap point data, the system labels the points with corresponding body part locations and solves the full-body motions. However, MoCap data often contains mislabeling, occlusion and positional errors, requiring extensive manual correction. To alleviate this burden, we introduce RoMo, a learning-based framework for robustly labeling and solving raw optical motion capture data. In the labeling stage, RoMo employs a divide-and-conquer strategy to break down the complex full-body labeling challenge into manageable subtasks: alignment, full-body segmentation and part-specific labeling. To utilize the temporal continuity of markers, RoMo generates marker tracklets using a K-partite graph-based clustering algorithm, where markers serve as nodes, and edges are formed based on positional and feature similarities. For motion solving, to prevent error accumulation along the kinematic chain, we introduce a hybrid inverse kinematic solver that utilizes joint positions as intermediate representations and adjusts the template skeleton to match estimated joint positions. We demonstrate that RoMo achieves high labeling and solving accuracy across multiple metrics and various datasets. Extensive comparisons show that our method outperforms state-of-the-art research methods. On a real dataset, RoMo improves the F1 score of hand labeling from 0.94 to 0.98, and reduces joint position error of body motion solving by 25%.

Introduction

Optical motion capture (MoCap) provides high-fidelity data essential for animation and biomechanics. However, the raw output—a cloud of unlabeled 3D points—is often corrupted by noise. Markers can be occluded, mislabeled, or jittery, necessitating significant manual cleanup. Existing automated solvers often struggle with dense marker clusters (like hands) and suffer from error accumulation along the kinematic chain.

Overview of the RoMo framework: (a) raw unlabeled markers, (b) labeling with outliers and occlusions, (c) iterative IK solving, and (d) the final solved motion.

RoMo (Robust Solver) is a learning-based framework designed to automate this process. It combines a "divide-and-conquer" labeling strategy with a hybrid inverse kinematics (IK) solver to handle complex scenarios that typically cause commercial systems to fail.

Methodology

The RoMo pipeline is divided into three primary stages: pre-processing, tracklet generation/labeling, and motion solving.

The RoMo pipeline consists of three main stages: point cloud pre-processing, tracklet generation and labeling, and IK-based motion solving.

Robust Labeling via Divide-and-Conquer

Instead of processing the entire point cloud at once, RoMo segments the data into body, left hand, and right hand components. This allows specialized sub-networks to handle the distinct motion patterns and marker densities of hands versus the body. To maintain temporal consistency, RoMo uses a K-partite graph-based clustering algorithm to link markers across frames into "tracklets." These tracklets are formed based on both 3D spatial proximity and deep features, ensuring robustness even during rapid movements or brief occlusions.

Hybrid IK Solving

Traditional solvers often estimate joint rotations sequentially, which can lead to "snowballing" errors. RoMo mitigates this by first estimating 3D joint positions as an intermediate representation. These global positions then guide a hybrid IK solver. By anchoring the skeleton to estimated global positions rather than relying solely on local rotations, RoMo prevents error accumulation and ensures the final motion remains anatomically plausible.

Results and Discussion

RoMo was evaluated on production-grade datasets and the public GRAB dataset with simulated noise. Key findings include:

  • Improved Accuracy: On real-world data, RoMo improved the F1 score for hand labeling from 0.94 to 0.98.
  • Reduced Error: It reduced the Mean Per Joint Position Error (MPJPE) for body motion solving by 25% compared to state-of-the-art methods like SOMA.
  • Robustness to Occlusion: The system maintained high performance even with 15% marker occlusion, a scenario where frame-by-frame methods typically fail.

Conclusion

RoMo represents a significant advancement in automated MoCap processing. By leveraging deep features for tracking and using joint positions as stable constraints for IK, it provides a robust solution for high-fidelity motion capture. Future work will focus on end-to-end integration of labeling and solving, as well as improving generalization across arbitrary marker layouts.

Summary updated

Abstract

Optical motion capture (MoCap) is the "gold standard" for accurately capturing full-body motions. To make use of raw MoCap point data, the system labels the points with corresponding body part locations and solves the full-body motions. However, MoCap data often contains mislabeling, occlusion and positional errors, requiring extensive manual correction. To alleviate this burden, we introduce RoMo, a learning-based framework for robustly labeling and solving raw optical motion capture data. In the labeling stage, RoMo employs a divide-and-conquer strategy to break down the complex full-body labeling challenge into manageable subtasks: alignment, full-body segmentation and part-specific labeling. To utilize the temporal continuity of markers, RoMo generates marker tracklets using a K-partite graph-based clustering algorithm, where markers serve as nodes, and edges are formed based on positional and feature similarities. For motion solving, to prevent error accumulation along the kinematic chain, we introduce a hybrid inverse kinematic solver that utilizes joint positions as intermediate representations and adjusts the template skeleton to match estimated joint positions. We demonstrate that RoMo achieves high labeling and solving accuracy across multiple metrics and various datasets. Extensive comparisons show that our method outperforms state-of-the-art research methods. On a real dataset, RoMo improves the F1 score of hand labeling from 0.94 to 0.98, and reduces joint position error of body motion solving by 25%. Furthermore, RoMo can be applied in scenarios where commercial systems are inadequate. The code and data for RoMo are available at https://github.com/non-void/RoMo.
BibTeX
@inproceedings{pan2024romo,
  author = {Xiaoyu Pan and Bowen Zheng and Xinwei Jiang and Zijiao Zeng and Qilong Kou and He Wang and Xiaogang Jin},
  title = {{RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture}},
  booktitle = {SIGGRAPH Asia},
  year = {2024},
  doi = {10.1145/3680528.3687615},
}
Figure