Abstract
Optical motion capture (mocap) requires accurately reconstructing the human body from retroreflective markers, including pose and shape. In a typical mocap setting, marker labeling is an important but tedious and error-prone step. Previous work has shown that marker labeling can be automated by using a structured template defining specific marker placements, but this places additional recording constraints. We propose to relax these constraints and solve for Unstructured Unlabeled Optical (UUO) mocap. Compared to the typical mocap setting that either labels markers or places them w.r.t a structured layout, markers in UUO mocap can be placed anywhere on the body and even on one specific limb (e.g., right leg for biomechanics research), hence it is of more practical significance. It is also more challenging. To solve UUO mocap, we exploit a monocular video captured by a single RGB camera, which does not require camera calibration. On this video, we run an off-the-shelf method to reconstruct and track a human individual, giving strong visual priors of human body pose and shape. With both the video and UUO markers, we propose an optimization pipeline towards marker identification, marker labeling, human pose estimation, and human body reconstruction. Experiments demonstrate that our method resoundingly outperforms existing methods on three established benchmark datasets for both full-body and partial-body reconstruction.
What Problem Does This Paper Address?
Optical motion capture (mocap) is the gold standard for tracking human movement, but it comes with a high "setup tax." Traditionally, you need a structured layout (markers must be placed in specific spots) and manual labeling (identifying which dot is the "left elbow").
While some automated labeling systems exist, they usually require a "template" of where markers should be. If a researcher wants to do a "partial body" study—for example, placing markers only on a subject's right leg to study gait—most automated systems fail because they lack the context of the rest of the body. The authors tackle the UUO Mocap problem: reconstructing motion from a cloud of 3D points where we don't know which point is which, and the points aren't placed in any specific pattern.
Methods
The core "aha!" moment of this paper is that while monocular video is too low-resolution for perfect mocap, it provides the "global context" that unlabeled 3D points lack.
The pipeline consists of three main modules:
- Monocular Reconstruction: The system uses an off-the-shelf tool (HMR2.0) to extract a rough 3D body model (SMPL) from a standard RGB video. This provides a "warm start" for pose and shape, though it often struggles with absolute scale and global orientation.
- Marker-Part Matching: To figure out which markers belong to which body part, the system groups markers that move rigidly together (using the standard deviation of distances between markers over time). It then matches these groups to "kinematic chains" (e.g., thigh-calf-foot) in the SMPL model.
- Mocap Solver: A multi-stage optimization process refines the fit.
- Multi-Hypothesis Testing: To avoid the common pitfall of the model facing the wrong way, the system tests four cardinal rotations (0°, 90°, 180°, 270°) and picks the best fit.
- Inverse Kinematics (IK): Finally, it "snaps" the body model to the markers to ensure high-precision tracking.
Key Insights and Ideas
- Video as a "Topology" Guide: Unlabeled 3D points are just a cloud of dots. Video provides the "connective tissue," telling the system that this cluster of dots is likely an arm because it's attached to a torso in the video frames.
- Rigidity is a Label: Even without labels, markers on the same bone maintain a constant distance from each other. This physical constraint allows the system to segment the marker cloud into body parts automatically.
- Solving the "Flip" Problem: Optimization often gets stuck in local minima where the body is rotated 180 degrees. By explicitly testing multiple rotation hypotheses at the start, the authors significantly improved reliability.
- Partial Body Flexibility: Because the system uses a full-body prior from video, it can accurately solve for a single limb (like just a leg) even when markers for the rest of the body are missing.
Key Results
The proposed method "resoundingly outperforms" existing state-of-the-art (SOTA) methods like SOMA and VPoser/HuMoR.
- Full-Body Accuracy: On the UMPM dataset, the method achieved a Mean Per-Joint Position Error (MPJPE) of 60.8mm, compared to SOMA's 101.1mm.
- Partial-Body Superiority: This is where the gap was widest. For "Left Leg" reconstruction, SOMA had an error of 678.1mm, while this method maintained a respectable 278.3mm.
- Robustness: The system remains stable even when markers are randomly dropped or placed in sub-optimal, "unstructured" locations.
Limitations and Future Work
- Symmetry Ambiguity: If a subject only has markers on one leg, the system might accidentally map the markers to the wrong leg during symmetrical movements.
- Sparse Layouts: Extremely sparse layouts still lead to labeling errors despite the visual prior.
- Environmental Constraints: The current model lacks floor awareness, which can occasionally lead to "foot sliding." Future versions may incorporate physical scene constraints.
Summary updated