TL;DR
RoMo (Robust Solver) is a new learning-based framework designed to clean and solve messy, unlabeled optical motion capture (MoCap) data. By combining a "divide-and-conquer" labeling strategy with a hybrid inverse kinematics (IK) solver that uses joint positions as intermediate steps, RoMo significantly outperforms existing methods in handling occlusions, mislabeled markers, and complex hand-body coordination.
What Problem Does This Paper Address?
Optical MoCap is the "gold standard" for high-fidelity animation, but the raw data it produces is notoriously "noisy." Markers—the little reflective balls on an actor's suit—frequently disappear (occlusion), get swapped with one another (mislabeling), or suffer from jitter (positional inaccuracy).
Cleaning this data usually requires hours of tedious manual labor. While automated solvers exist, they often struggle with two specific areas:
- Full-body Integration: Most solvers handle the body well but fail on the dense, high-speed marker clusters of the hands.
- Error Accumulation: Traditional solvers estimate joint rotations sequentially. If the shoulder rotation is slightly off, that error "snowballs" down the arm, leading to massive positional errors at the fingertips.
Methods
RoMo tackles these issues through a two-stage pipeline:
1. Robust Labeling (The "Divide-and-Conquer" Stage)
Instead of throwing the entire point cloud into one neural network, RoMo breaks the task down:
- Alignment & Segmentation: It first aligns the point cloud to a standard orientation and segments it into three parts: body, left hand, and right hand. This allows specialized sub-networks to handle the distinct data distributions of hands vs. bodies.
- Tracklet Generation: To leverage temporal information, RoMo uses a K-partite graph-based clustering algorithm. It links markers across frames into "tracklets" based on both their 3D positions and deep features extracted by the network. This ensures that even if a marker is briefly occluded, the system remembers which label it belongs to.
2. Hybrid IK Solving (The "Global Awareness" Stage)
To solve for the actual skeletal motion, RoMo avoids predicting rotations directly from the start.
- Intermediate Representation: The network first estimates 3D joint positions.
- Hybrid IK: It then uses these global positions to guide an Inverse Kinematics solver. By anchoring the skeleton to estimated global positions rather than just local rotations, RoMo prevents error accumulation along the kinematic chain.
Datasets
The authors evaluated RoMo on three primary datasets:
- Production & Front-waist Datasets: High-quality, professionally captured and hand-cleaned datasets from a game studio, featuring different marker layouts.
- GRAB Dataset: A public dataset of humans interacting with objects. The authors added simulated noise (up to 15% occlusion) to test RoMo’s limits in "limited hardware" scenarios.
Key Insights and Ideas
- Feature-Based Tracking: Most tracklet methods rely only on position. RoMo uses "deep features" (learned characteristics of the marker's neighborhood), which makes the tracking much more resilient during rapid movements where markers might overlap in 3D space.
- Positions Over Rotations: The decision to use joint positions as an intermediate step is a major "aha moment." Rotations are non-linear and hard for networks to learn directly; positions provide a stable, global constraint that keeps the skeleton from drifting.
- Specialization Matters: Treating hands as a separate segmentation task acknowledges that hand markers are closer together and move much faster than body markers, requiring a different "resolution" of analysis.
Key Results
RoMo demonstrates a significant leap over state-of-the-art methods like SOMA and LocalMoCap:
- Labeling Accuracy: On real-world datasets, RoMo improved the F1 score for hand labeling from 0.94 to 0.98.
- Solving Precision: It reduced the joint position error (MPJPE) for body motion solving by 25%.
- Robustness: Even with 15% of markers missing (occlusion), RoMo maintained high labeling accuracy where previous frame-by-frame methods failed.
- Qualitative Success: The system successfully solved complex, multi-fingered hand movements and large body motions that typically cause commercial systems to glitch.
Limitations and Future Work
Despite its robustness, RoMo has a few areas for improvement:
- Two-Stage Disconnect: Currently, labeling and solving are separate steps. The authors suggest that merging these into a single end-to-end network could further improve performance.
- Layout Sensitivity: RoMo is still somewhat tied to the marker layout it was trained on. Training a single model that can generalize across any arbitrary marker layout remains an open challenge.
- Temporal Integration: While tracklets use temporal data, the labeling itself is still largely driven by spatial features. Integrating more complex motion priors into the labeling stage could help in extremely high-occlusion scenarios.
Summary updated