A Simple Method to Boost Human Pose Estimation Accuracy by Correcting the Joint Regressor for the Human3.6m Dataset

Eric Hedlin, Helge Rhodin, Kwang Moo Yi

Introduction

In the field of 3D human pose estimation, the Skinned Multi-Person Linear (SMPL) model has become the industry standard for representing the human form. By utilizing a detailed 3D mesh consisting of thousands of vertices, SMPL allows researchers to capture complex body shapes and poses. However, a significant discrepancy exists between these high-fidelity mesh outputs and the benchmarks used to evaluate them. Most major datasets, such as the widely used Human3.6m, evaluate performance based on a sparse set of skeletal joints rather than the full mesh.

To bridge this gap, researchers employ a joint regressor—a linear transformation layer that maps specific mesh vertices to skeletal joint locations (e.g., the hip, knee, or ankle). Recent findings from the University of British Columbia suggest that the primary bottleneck in state-of-the-art (SOTA) model performance may not be the models themselves, but rather inaccuracies within this regressor layer.

The Problem: The Regressor Misalignment

The standard joint regressor for the Human3.6m dataset was historically derived using a process called MoSh (Motion and Shape capture). A critical flaw in this process is that it relied on sparse markers and ignored the original images during the creation of the ground-truth mesh. This led to a systematic "misalignment": while a model might predict a 3D body shape that looks correct in isolation, the regressed joints often fail to align with the actual person in the 2D image.

Because benchmarks like Human3.6m calculate the Mean Per Joint Position Error (MPJPE) based on these potentially misaligned joint definitions, even a perfect 3D reconstruction can be penalized. This suggests that current SOTA models are likely more accurate than their reported metrics indicate, but they are being measured against a flawed yardstick.

Methodology: Optimizing the Joint Regressor

The researchers proposed a method to generate high-quality "pseudo-ground-truth" meshes for the Human3.6m dataset, which are then used to train a more accurate joint regressor ($J_{opt}$). Their approach optimizes the SMPL parameters (pose $\theta$ and shape $\beta$) of an existing model (SPIN) by minimizing a multi-term energy function:

  1. Ground-truth Joint Alignment ($L_{joint}$): This term ensures the regressed joints remain consistent with the 3D coordinates provided by the dataset's motion capture system.
  2. Silhouette Alignment ($L_{silhouette}$): This is the core innovation. By using Mask R-CNN to extract silhouettes from the original images and a differentiable renderer to project the 3D mesh back into 2D, the authors optimize the mesh so its projection perfectly fits the person's visual outline. This ensures the 3D model is physically constrained by the image evidence.
  3. Adversarial Plausibility ($L_{adv}$): To ensure that the optimization does not result in anatomically impossible poses (often a risk when fitting to 2D silhouettes), a discriminator network is used to keep the poses within the distribution of realistic human movement.

After generating these optimized meshes, a new linear joint regressor is trained using non-negative least squares, providing a more faithful mapping from mesh vertices to joint centers.

Results and Impact

The beauty of this approach is its "plug-and-play" nature. Because the joint regressor is a simple linear layer at the end of the inference pipeline, the improved regressor can be applied to existing models without retraining the core neural networks. The authors demonstrated immediate improvements in MPJPE across several leading models:

  • SPIN: Error reduced from 60.4mm to 56.9mm.
  • VIBE: Error reduced from 62.9mm to 60.0mm.
  • MEVA: Error reduced from 101.6mm to 98.3mm.

Qualitative analysis confirms that the new regressor places joints—particularly at the extremities like the feet and ankles—much more accurately within the visible bounds of the human body in the source images.

Conclusion and Future Work

This research highlights a critical oversight in how 3D human pose estimation is evaluated. By correcting the joint regressor, the authors have shown that significant accuracy gains can be achieved simply by ensuring our evaluation tools match the visual reality of the datasets.

While this method provides a significant boost, challenges remain regarding label inconsistency, where different annotators may define the "center" of a joint differently. Future research may focus on integrating these silhouette and plausibility constraints directly into the training loops of pose estimation models, rather than applying them as a post-hoc correction to the evaluation layer. For now, this refined regressor offers a more accurate lens through which to measure the progress of 3D human reconstruction.

Summary updated

Abstract

Many human pose estimation methods estimate Skinned Multi-Person Linear (SMPL) models and regress the human joints from these SMPL estimates. In this work, we show that the most widely used SMPL-to-joint linear layer (joint regressor) is inaccurate, which may mislead pose evaluation results. To achieve a more accurate joint regressor, we propose a method to create pseudo-ground-truth SMPL poses, which can then be used to train an improved regressor. Specifically, we optimize SMPL estimates coming from a state-of-the-art method so that its projection matches the silhouettes of humans in the scene, as well as the ground-truth 2D joint locations. While the quality of this pseudo-ground-truth is chal-lenging to assess due to the lack of actual ground-truth SMPL, with the Human 3.6m dataset, we qualitatively show that our joint locations are more accurate and that our regressor leads to improved pose estimations results on the test set without any need for retraining. We release our code and joint regressor at https://github.com/ubc-vision/joint-regressor-refinement
BibTeX
@inproceedings{hedlin2022simple,
  author = {Eric Hedlin and Helge Rhodin and Kwang Moo Yi},
  title = {{A Simple Method to Boost Human Pose Estimation Accuracy by Correcting the Joint Regressor for the Human3.6m Dataset}},
  booktitle = {CRV},
  year = {2022},
  url = {https://openreview.net/forum?id=TI4k0URfIYy},
}
Figure