DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Camera

Shaohua Pan, Xinyu Yi, Yan Zhou, Weihua Jian, Yuan Zhang, Pengfei Wan, Feng Xu

TL;DR

This paper introduces DiffCap, a novel diffusion-based approach for real-time human motion capture using sparse IMUs and a single camera. By intelligently fusing these two data sources, DiffCap achieves state-of-the-art accuracy and robustness, even in challenging scenarios like occlusions and out-of-view situations.

What Problem Does This Paper Address?

Accurate and convenient human motion capture (mocap) is crucial for fields like VR, animation, and sports. While professional systems exist, they are expensive and complex. Researchers are exploring using everyday sensors like cameras and IMUs. Existing methods combining these sensors struggle with:

  • Vision-based methods: Fail when visual data is compromised (occlusions, out-of-frame subjects).
  • IMU-based methods: Suffer from drift and pose ambiguity due to sparse sensors.
  • Fusion methods: Often rely on complex optimization, struggle with efficiency, or are sensitive to the accuracy of 2D keypoint detection and require careful threshold tuning for switching between modalities.

The core challenge is effectively fusing signals with time-varying confidences – IMUs are generally stable but can drift over time, while camera data is rich but prone to unpredictable occlusions.

Methods

DiffCap tackles this by leveraging diffusion models, known for their generative capabilities and ability to learn data priors. The approach has two main stages:

  1. Joint Diffusion Model: This model estimates intermediate 3D joint positions. It takes sequential 2D keypoints (processed to be root-relative and conditioned on confidence) and IMU measurements as input. The 2D keypoints are transformed into a single "condition embedding" to make the model robust to occasional visual dropouts. IMU data, being more stable, is concatenated frame-wise with the noised joint positions to preserve temporal information.
  2. Pose Diffusion Model: This model refines the 3D joint positions into full human poses (SMPL parameters). It uses the estimated 3D joint positions from the first stage, IMU data, and the noised pose rotations as input. Notably, it excludes 2D keypoints at this stage, relying on the IMU data as a robust, sequential input.

The core innovation lies in how the two modalities are fused: visual information is treated as a global condition, robust to temporary unavailability, while IMU data is used as a frame-wise sequential input to capture local temporal dynamics. This design choice is motivated by the observation that visual information can be unreliable, while IMUs are generally stable.

Datasets

  • Training: AMASS dataset and AIST++ dataset. For AIST++, they used detected 2D keypoints and synthesized IMUs. For AMASS, they used synthesized 2D keypoints and IMUs.
  • Evaluation: AIST++ test split, TotalCapture, 3DPW, and 3DPW-OCC. For evaluation, they used synthesized IMUs.

Key Insights and Ideas

  • Global Visual Conditioning for Robustness: Treating sequential visual features (2D keypoints) as a single, global condition embedding makes the model resilient to occlusions or subjects moving out of frame. This is a key differentiator from methods that rely on per-frame visual confidence.
  • Frame-wise IMU for Temporal Detail: Concatenating IMU measurements frame-by-frame with the diffusion model's input preserves their temporal ordering and allows for richer exploitation of local motion dynamics, especially when visual data is unreliable.
  • Two-Stage Diffusion for Complexity Management: Decomposing the problem into estimating joint positions first, then poses, simplifies the task and leads to improved performance and real-time applicability.
  • Diffusion Models as Priors for Reconstruction: The paper demonstrates that diffusion models, typically used for generation, are also highly effective for reconstruction tasks like motion capture, by learning strong human motion priors.

Key Results

  • State-of-the-Art Performance: DiffCap achieves state-of-the-art results on multiple challenging datasets (3DPW, 3DPW-OCC, AIST++, TotalCapture) across various metrics (MPJPE, PA-MPJPE, PVE). For example, on 3DPW, it achieves an MPJPE of 46.9mm, outperforming previous fusion methods.
  • Robustness to Occlusions and Out-of-View: Qualitative results show DiffCap performing significantly better than vision-based methods when subjects are out of view and handling occlusions more gracefully than other fusion methods.
  • Real-time Capability: The method is designed for real-time performance, demonstrated by live demos.
  • Reduced Sensitivity to Noise: DiffCap is shown to be less sensitive to noise in 2D keypoints compared to RobustCap, indicating its more robust fusion strategy.

Limitations and Future Work

  • Translation Estimation: The current method focuses on pose estimation and does not explicitly improve translation estimation, which is suggested as a promising future direction.
  • Magnetic Disturbances: IMU data can be affected by magnetic disturbances. Modeling and mitigating these disturbances is an open question.
  • Non-physical Artifacts: The current kinematic-based approach might produce non-physical artifacts like jitter. Future work could explore physical optimization or reinforcement learning to address this.

Summary updated

Abstract

Combining sparse IMUs and a monocular camera is a new promising setting to perform real-time human motion capture. This paper proposes a diffusion-based solution to learn human motion priors and fuse the two modalities of signals together seamlessly in a unified framework. By delicately considering the characteristics of the two signals, the sequential visual information is considered as a whole and transformed into a condition embedding, while the inertial measurement is concatenated with the noisy body pose frame by frame to construct a sequential input for the diffusion model. Firstly, we observe that the visual information may be unavailable in some frames due to occlusions or subjects moving out of the camera view. Thus incorporating the sequential visual features as a whole to get a single feature embedding is robust to the occasional degenerations of visual information in those frames. On the other hand, the IMU measurements are robust to occlusions and always stable when signal transmission has no problem. So incorporating them frame-wisely could better explore the temporal information for the system. Experiments have demonstrated the effectiveness of the system design and its state-of-the-art performance in pose estimation compared with the previous works. Our codes are available for research at https://shaohua-pan.github.io/diffcap-page.
BibTeX
@article{pan2025diffcap,
  author = {Shaohua Pan and Xinyu Yi and Yan Zhou and Weihua Jian and Yuan Zhang and Pengfei Wan and Feng Xu},
  title = {{DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Camera}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
}
Figure