Multi-View 3D Human Pose Estimation with Weakly Synchronized Images

Ling Li, Ruiwen Gu, Chongyang Wang, Junliang Xing, Xinchun Yu, Xiao-Ping ZhangAAAI 2025 · DOI

TL;DR

This paper tackles the challenge of 3D human pose estimation from multiple camera views when those cameras aren't perfectly synchronized. They introduce a new task and dataset for "weakly synchronized images" and propose SyncDiffPose, a diffusion-based model that effectively denoises temporal inconsistencies to achieve accurate pose estimation without the need for expensive hardware synchronization.

What Problem Does This Paper Address?

Estimating the 3D pose of a human from multiple camera views (Multi-View 3D Human Pose Estimation or MHPE) is crucial for applications like sports analysis and motion capture. Traditionally, MHPE relies on cameras being perfectly synchronized to ensure that images from different views capture the same moment in time. This synchronization is typically achieved using specialized hardware, which is expensive and inflexible, especially for outdoor scenarios. While camera-based clock synchronization exists, it's complex and susceptible to network fluctuations.

The core problem this paper addresses is the limitations imposed by this strict synchronization requirement. They propose that MHPE models should be robust to slight temporal misalignments between camera views, a scenario they term "weakly synchronized images." This would significantly reduce deployment costs and broaden the applicability of MHPE. The challenge lies in fusing information from these slightly desynchronized views without the temporal inconsistencies introducing significant errors, particularly for fast-moving body parts.

Methods

The paper introduces a novel approach called SyncDiffPose, which leverages the power of diffusion models for pose estimation. Diffusion models are generative models that work by gradually adding noise to data and then learning to reverse this process to generate clean data.

Here's a breakdown of their method:

  • Weakly Synchronized Images: They define this as a set of multi-view images where the time difference between corresponding frames from different cameras is within a single frame interval (i.e., less than 1/f, where f is the camera frame rate).
  • SyncDiffPose Architecture:
    • Spatial Encoder: This component takes the weakly synchronized 2D poses from multiple views and extracts a unified spatial feature representation.
  • Noise Embedding (NE) Module: This novel module is designed to specifically model the noise introduced by weak temporal synchronization. It learns to map the input pose to a noisy distribution that reflects these temporal discrepancies.
  • Noise Estimation Model (NEM): This is the core diffusion model. It iteratively denoises the noisy 3D pose distribution generated by the NE module. The NEM uses a sequence-to-sequence architecture with self-attention and cross-attention mechanisms.
  • Reverse Process: The NEM progressively removes noise over multiple steps, guided by the diffusion step and the spatial features, to reconstruct a clean 3D pose.

The key innovation is the integration of the NE module to explicitly model and learn from the specific type of noise introduced by weakly synchronized images, allowing the diffusion process to effectively denoise these errors.

Datasets

  • WeakSyncPose3D: The authors introduce this as the first large-scale dataset specifically for weakly synchronized multi-view 3D human pose estimation. It contains approximately 800,000 frames of action with high motion frequencies, captured using a setup with one synchronized optical camera and multiple weakly synchronized motion capture cameras. Ground truth 3D joint coordinates are provided by OptiTrack devices.
  • Human3.6M: This is a widely used, well-synchronized multi-view 3D human pose dataset. The authors use it for evaluation, including testing their model on data with a simulated one-frame frameshift to mimic weakly synchronized conditions.

Key Insights and Ideas

  • The "Weakly Synchronized" Concept: Defining and formalizing the problem of MHPE with images that are almost synchronized but not perfectly is a crucial step. This opens up a new avenue of research beyond strictly synchronized data.
  • Diffusion Models for Denoising Temporal Errors: Applying diffusion models, which are excellent at generative tasks and denoising, to the problem of temporal inconsistencies in MHPE is a powerful idea. The diffusion process naturally lends itself to iteratively refining pose estimates by removing noise.
  • Dedicated Noise Embedding Module: Creating a specific module (NE) to learn the characteristics of noise introduced by weak synchronization is a clever way to make the diffusion model more effective for this particular challenge. It allows the model to better understand and correct the specific types of errors encountered.
  • Improved Performance on Weakly Synchronized Data: The paper highlights that their model performs even better on weakly synchronized data compared to synchronized data from Human3.6M. This suggests that the model's ability to handle temporal noise is a significant advantage, potentially even outperforming synchronized methods when synchronization is imperfect.

Key Results

  • WeakSyncPose3D Dataset: The creation of this dataset is a significant contribution, providing a benchmark for future research in this area.
  • SyncDiffPose Performance: The proposed SyncDiffPose model achieves promising results on both the WeakSyncPose3D and Human3.6M datasets.
  • Quantitative Improvements: While specific MPJPE (Mean Per-Joint Position Error) numbers are presented in tables, the paper emphasizes that their model shows "even more significant improvements on the weakly synchronized dataset" compared to synchronized data. Table 2 shows a reduction in MPJPE on Human3.6M with a one-frame frameshift compared to the original synchronized data when using their model.
  • Qualitative Improvements: Figure 5 visually demonstrates the superior performance of SyncDiffPose, showing more accurate and coherent 3D human poses compared to other single-view and multi-view methods, especially in challenging scenarios.

Limitations and Future Work

  • Synchronization Capabilities: The authors acknowledge that despite achieving commendable results, their model's performance is still constrained by insufficient synchronization capabilities. It cannot reach the optimal levels observed in models that utilize perfectly synchronized images as input.
  • Future Directions: The paper implicitly suggests that further improvements in modeling synchronization errors and potentially exploring more advanced diffusion techniques could lead to even better performance. The development of more diverse and challenging weakly synchronized datasets would also be beneficial.

Summary updated

BibTeX
@inproceedings{li2025multi,
  author = {Ling Li and Ruiwen Gu and Chongyang Wang and Junliang Xing and Xinchun Yu and Xiao-Ping Zhang},
  title = {{Multi-View 3D Human Pose Estimation with Weakly Synchronized Images}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32511},
}
Figure