Sim2real transfer learning for 3D human pose estimation: motion to the rescue

Carl Doersch, Andrew Zisserman

Deep learning models for 3D human pose estimation are notoriously "data-hungry." However, obtaining ground-truth 3D labels for real-world images is incredibly difficult, often requiring expensive motion-capture (MoCap) setups or intrusive sensors that do not work "in the wild."

Synthetic data (like the SURREAL dataset) offers a tempting solution by providing infinite, perfectly labeled data without privacy concerns. The catch is the sim2real gap. Neural networks trained on synthetic RGB images often overfit to the specific textures, lighting, and "perfect" renders of the simulator, causing them to fail miserably when faced with the messy complexity of real-world video.

This paper demonstrates that motion cues—specifically optical flow and 2D keypoint tracks—are the "missing link" for bridging this sim2real gap. By training on synthetic humans but using motion-based inputs, the authors achieve state-of-the-art performance on real-world datasets, proving that temporal dynamics are far more robust to domain shifts than static RGB textures.

The Motion HMR Framework

To address the sim2real gap, the authors introduce Motion HMR, an evolution of the standard Human Mesh Recovery (HMR) framework. Their approach relies on three key pillars:

Pipeline overview showing Input, Flow, Keypoints, and 3D Estimates
  1. Motion-Centric Inputs: Instead of relying solely on RGB frames, the model takes a stack of optical flow (computed via FlowNet) and 2D keypoint heatmaps. The intuition is that while a synthetic person's skin texture looks "fake," the way their elbow moves relative to their shoulder is geometrically consistent with a real human.
  2. Temporal Integration: They augment the HMR architecture with a bi-directional LSTM. This allows the model to integrate information over 31-frame clips, helping it resolve ambiguities (like occlusions) by looking at how a person moved before and after a specific moment.
  3. Realistic Synthetic Training: To make the synthetic data more challenging, they developed a sophisticated compositing pipeline. They took synthetic humans from SURREAL and placed them into real-world background videos from the Kinetics dataset. Crucially, they matched the camera motion of the background and used SLIC superpixels to create "natural" occlusions, simulating objects passing in front of the subject.

Key Insights and Ideas

  • RGB is a Distractor: One of the most surprising findings was that adding RGB information to the motion features actually decreased performance. The network would latch onto low-level synthetic textures (which don't exist in the real world) and ignore the more reliable geometric motion cues.
  • Motion as a Universal Language: Optical flow and 2D keypoints act as an abstraction layer. They strip away domain-specific "noise" (like lighting and clothing patterns) and preserve the underlying physics and geometry of human movement, which is the same in both simulation and reality.
  • The Importance of Occlusion: Simply sticking a synthetic person on a background isn't enough. By using superpixels from the background video to "cut out" parts of the synthetic human, the authors forced the model to learn how to "fill in the blanks" for hidden limbs—a vital skill for real-world performance.

Datasets and Key Results

The model was trained using SURREAL for synthetic human models and 3D ground truth, paired with Kinetics to provide realistic, moving backgrounds. The primary evaluation benchmark was 3D Poses in the Wild (3DPW), a highly challenging dataset consisting of real-world video of people outdoors, labeled using IMU sensors.

The primary metric used is PA-MPJPE (Reconstruction Error in mm, where lower is better):

  • RGB-only (Synthetic training): 105.6 mm. This shows the massive failure of naive sim2real transfer.
  • Motion HMR (Flow + Keypoints, Synthetic training): 74.7 mm.
  • Comparison: For context, the original HMR model trained on real 3D data achieves roughly 77.2 mm on the same benchmark.

This is a landmark result: a model trained entirely on synthetic humans outperformed a standard model trained on real-world 3D sequences.

Limitations and Future Work

While the results are impressive, the authors note a few hurdles:

  • LSTM Accumulation: Performance peaked at 31-frame sequences; longer sequences (e.g., 2 seconds) didn't help, suggesting that errors might accumulate in the LSTM over time or that batch size limitations (due to memory) hindered training.
  • Reliance on 2D Detectors: The system still requires a robust 2D keypoint detector and optical flow algorithm. While these are generally reliable, failures in these "upstream" components will propagate to the 3D estimate.
  • Untapped RGB Potential: The authors suggest that while raw RGB was a distractor here, future work could use self-supervised learning to "re-introduce" RGB cues in a way that doesn't cause overfitting to synthetic textures.

Summary updated

Abstract

Synthetic visual data can provide practically infinite diversity and rich labels, while avoiding ethical issues with privacy and bias. However, for many tasks, current models trained on synthetic data generalize poorly to real data. The task of 3D human pose estimation is a particularly interesting example of this sim2real problem, because learning-based approaches perform reasonably well given real training data, yet labeled 3D poses are extremely difficult to obtain in the wild, limiting scalability. In this paper, we show that standard neural-network approaches, which perform poorly when trained on synthetic RGB images, can perform well when the data is pre-processed to extract cues about the person's motion, notably as optical flow and the motion of 2D keypoints. Therefore, our results suggest that motion can be a simple way to bridge a sim2real gap when video is available. We evaluate on the 3D Poses in the Wild dataset, the most challenging modern benchmark for 3D pose estimation, where we show full 3D mesh recovery that is on par with state-of-the-art methods trained on real 3D sequences, despite training only on synthetic humans from the SURREAL dataset.
BibTeX
@inproceedings{doersch2019sim,
  author = {Carl Doersch and Andrew Zisserman},
  title = {{Sim2real transfer learning for 3D human pose estimation: motion to the rescue}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure