DiffPose: Multi-hypothesis Human Pose Estimation using Diffusion Models

Karl Holmquist, Bastian Wandt

Abstract

Traditionally, monocular 3D human pose estimation employs a machine learning model to predict the most likely 3D pose for a given input image. However, a single image can be highly ambiguous and induces multiple plausible solutions for the 2D-3D lifting step, which results in overly confident 3D pose predictors. To this end, we propose DiffPose, a conditional diffusion model that predicts multiple hypotheses for a given input image. Compared to similar approaches, our diffusion model is straightforward and avoids intensive hyperparameter tuning, complex network structures, mode collapse, and unstable training. Moreover, we tackle the problem of over-simplification of the intermediate representation of the common two-step approaches which first estimate a distribution of 2D joint locations via joint-wise heatmaps and consecutively use their maximum argument for the 3D pose estimation step. Since such a simplification of the heatmaps removes valid information about possibly correct, though labeled unlikely, joint locations, we propose to represent the heatmaps as a set of 2D joint candidate samples. To extract information about the original distribution from these samples, we introduce our embedding transformer which conditions the diffusion model. Experimentally, we show that DiffPose improves upon the state of the art for multi-hypothesis pose estimation by 3-5% for simple poses and outperforms it by a large margin for highly ambiguous poses.

Introduction

Estimating a 3D human pose from a single 2D image is an inherently "ill-posed" problem. Due to the lack of depth information in monocular images, a single 2D projection can correspond to numerous valid 3D configurations. This ambiguity is exacerbated in real-world scenarios where body parts are occluded or the camera angle is challenging.

Most traditional models attempt to regress a single "most likely" 3D pose. This approach often leads to overly confident but physically incorrect predictions. While multi-hypothesis methods have been proposed to address this, they frequently suffer from two major drawbacks: oversimplification of 2D joint probability maps (heatmaps) into single points, and training instabilities common in generative models like GANs or VAEs. DiffPose addresses these issues by leveraging the stability and expressive power of Denoising Diffusion Probabilistic Models (DDPMs).

Methodology

DiffPose treats 3D pose estimation as a conditional generation task. The framework consists of a diffusion process conditioned on rich 2D joint information extracted from heatmaps.

The Diffusion Process

The model learns to reverse a forward process that gradually adds Gaussian noise to a ground-truth 3D pose. During inference, the model starts with pure noise and iteratively refines it into a realistic 3D pose, guided by the input 2D conditioning data. This iterative refinement naturally enforces anatomical constraints, such as bone lengths and symmetry, without requiring explicit physical rules.

Heatmap Sampling and Embedding Transformer

Instead of reducing 2D heatmaps to their maximum values, DiffPose draws $n$ samples from the distribution. This preserves the full shape of the uncertainty, including multi-modal distributions where a joint might plausibly exist in multiple distinct locations.

The DiffPose framework: 2D heatmaps are sampled and processed by an Embedding Transformer to condition a Denoising Diffusion Probabilistic Model (DDPM).

To process these samples, the authors developed an Embedding Transformer. This architecture takes the 2D samples and their associated confidence scores, encodes them using non-linear "channel embeddings," and employs self-attention to model the spatial relationships between different joints. This conditioned representation then guides the neural network (NN) within the diffusion loop.

Key Results and Insights

DiffPose demonstrates significant improvements over existing state-of-the-art (SOTA) methods, particularly in cases of high ambiguity.

  • Superiority in Ambiguity: On the H36MA dataset—a subset of Human3.6M specifically curated for high uncertainty—DiffPose outperformed previous SOTA methods by 9% in MPJPE and 12% in PA-MPJPE.
  • Physical Plausibility: The model achieved a 46% lower symmetry error compared to competitors, indicating that the generated poses are more anatomically consistent.
  • Diversity and Accuracy: By generating a diverse set of hypotheses (e.g., up to 200), the model is significantly more likely to capture the true ground truth within its prediction set than single-shot models.
Qualitative comparison of DiffPose against state-of-the-art methods on an ambiguous pose, highlighting the model's ability to generate multiple plausible 3D hypotheses.

As shown in the qualitative comparison, DiffPose effectively captures the uncertainty of joint positions (like the right wrist) through its heatmap sampling and multi-hypothesis generation, whereas other methods often collapse to a single, incorrect estimate.

Conclusion and Future Work

DiffPose proves that diffusion models are exceptionally well-suited for modeling the complex manifold of human skeletons. By preserving non-Gaussian information from 2D heatmaps, the model avoids the information loss typical of traditional two-step approaches.

However, limitations remain. As a "lifting" method, DiffPose is dependent on the quality of the initial 2D joint detector. Additionally, the iterative nature of diffusion makes inference slower than single-shot regression models. Future research may focus on incorporating temporal data from video sequences to further resolve depth ambiguities or integrating the diffusion process directly with raw image pixels.

Summary updated

Abstract

Traditionally, monocular 3D human pose estimation employs a machine learning model to predict the most likely 3D pose for a given input image. However, a single image can be highly ambiguous and induces multiple plausible solutions for the 2D-3D lifting step, which results in overly confident 3D pose predictors. To this end, we propose DiffPose, a conditional diffusion model that predicts multiple hypotheses for a given input image. Compared to similar approaches, our diffusion model is straightforward and avoids intensive hyperparameter tuning, complex network structures, mode collapse, and unstable training. Moreover, we tackle the problem of over-simplification of the intermediate representation of the common two-step approaches which first estimate a distribution of 2D joint locations via joint-wise heatmaps and consecutively use their maximum argument for the 3D pose estimation step. Since such a simplification of the heatmaps removes valid information about possibly correct, though labeled unlikely, joint locations, we propose to represent the heatmaps as a set of 2D joint candidate samples. To extract information about the original distribution from these samples, we introduce our embedding transformer which conditions the diffusion model. Experimentally, we show that DiffPose improves upon the state of the art for multi-hypothesis pose estimation by 3-5% for simple poses and outperforms it by a large margin for highly ambiguous poses.1
BibTeX
@inproceedings{holmquist2023diffpose,
  author = {Karl Holmquist and Bastian Wandt},
  title = {{DiffPose: Multi-hypothesis Human Pose Estimation using Diffusion Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=hK72VCrJPc},
}
Figure