TL;DR
This paper introduces DiffCap, a novel diffusion-based approach for real-time human motion capture using sparse IMUs and a single camera. By intelligently fusing these two data sources, DiffCap achieves state-of-the-art accuracy and robustness, even in challenging scenarios like occlusions and out-of-view situations.
What Problem Does This Paper Address?
Accurate and convenient human motion capture (mocap) is crucial for fields like VR, animation, and sports. While professional systems exist, they are expensive and complex. Researchers are exploring using everyday sensors like cameras and IMUs. Existing methods combining these sensors struggle with:
- Vision-based methods: Fail when visual data is compromised (occlusions, out-of-frame subjects).
- IMU-based methods: Suffer from drift and pose ambiguity due to sparse sensors.
- Fusion methods: Often rely on complex optimization, struggle with efficiency, or are sensitive to the accuracy of 2D keypoint detection and require careful threshold tuning for switching between modalities.
The core challenge is effectively fusing signals with time-varying confidences – IMUs are generally stable but can drift over time, while camera data is rich but prone to unpredictable occlusions.
Methods
DiffCap tackles this by leveraging diffusion models, known for their generative capabilities and ability to learn data priors. The approach has two main stages:
- Joint Diffusion Model: This model estimates intermediate 3D joint positions. It takes sequential 2D keypoints (processed to be root-relative and conditioned on confidence) and IMU measurements as input. The 2D keypoints are transformed into a single "condition embedding" to make the model robust to occasional visual dropouts. IMU data, being more stable, is concatenated frame-wise with the noised joint positions to preserve temporal information.
- Pose Diffusion Model: This model refines the 3D joint positions into full human poses (SMPL parameters). It uses the estimated 3D joint positions from the first stage, IMU data, and the noised pose rotations as input. Notably, it excludes 2D keypoints at this stage, relying on the IMU data as a robust, sequential input.
The core innovation lies in how the two modalities are fused: visual information is treated as a global condition, robust to temporary unavailability, while IMU data is used as a frame-wise sequential input to capture local temporal dynamics. This design choice is motivated by the observation that visual information can be unreliable, while IMUs are generally stable.
Datasets
- Training: AMASS dataset and AIST++ dataset. For AIST++, they used detected 2D keypoints and synthesized IMUs. For AMASS, they used synthesized 2D keypoints and IMUs.
- Evaluation: AIST++ test split, TotalCapture, 3DPW, and 3DPW-OCC. For evaluation, they used synthesized IMUs.
Key Insights and Ideas
- Global Visual Conditioning for Robustness: Treating sequential visual features (2D keypoints) as a single, global condition embedding makes the model resilient to occlusions or subjects moving out of frame. This is a key differentiator from methods that rely on per-frame visual confidence.
- Frame-wise IMU for Temporal Detail: Concatenating IMU measurements frame-by-frame with the diffusion model's input preserves their temporal ordering and allows for richer exploitation of local motion dynamics, especially when visual data is unreliable.
- Two-Stage Diffusion for Complexity Management: Decomposing the problem into estimating joint positions first, then poses, simplifies the task and leads to improved performance and real-time applicability.
- Diffusion Models as Priors for Reconstruction: The paper demonstrates that diffusion models, typically used for generation, are also highly effective for reconstruction tasks like motion capture, by learning strong human motion priors.
Key Results
- State-of-the-Art Performance: DiffCap achieves state-of-the-art results on multiple challenging datasets (3DPW, 3DPW-OCC, AIST++, TotalCapture) across various metrics (MPJPE, PA-MPJPE, PVE). For example, on 3DPW, it achieves an MPJPE of 46.9mm, outperforming previous fusion methods.
- Robustness to Occlusions and Out-of-View: Qualitative results show DiffCap performing significantly better than vision-based methods when subjects are out of view and handling occlusions more gracefully than other fusion methods.
- Real-time Capability: The method is designed for real-time performance, demonstrated by live demos.
- Reduced Sensitivity to Noise: DiffCap is shown to be less sensitive to noise in 2D keypoints compared to RobustCap, indicating its more robust fusion strategy.
Limitations and Future Work
- Translation Estimation: The current method focuses on pose estimation and does not explicitly improve translation estimation, which is suggested as a promising future direction.
- Magnetic Disturbances: IMU data can be affected by magnetic disturbances. Modeling and mitigating these disturbances is an open question.
- Non-physical Artifacts: The current kinematic-based approach might produce non-physical artifacts like jitter. Future work could explore physical optimization or reinforcement learning to address this.
Summary updated