Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation

Krishna Kanth Nakka, Mathieu SalzmannarXiv 2023 · DOI · arXiv

Introduction

Self-supervised 3D human pose estimation (HPE) has emerged as a powerful alternative to supervised learning, which requires expensive and difficult-to-obtain 3D annotations for "in-the-wild" images. These self-supervised methods rely on a fundamental assumption: that a model can perfectly "disentangle" a person’s physical pose from their visual appearance within its latent memory. This paper critically examines this assumption, demonstrating that current state-of-the-art methods fail to achieve true disentanglement, leading to models that are brittle and easily confused by changes in clothing or lighting.

Methodology

The standard self-supervised strategy involves training an encoder to split an image into two distinct vectors: a pose code (representing body geometry) and an appearance code (representing the subject's visual features). Ideally, these codes should be independent. To test this, the authors evaluated three major frameworks—NSD, CSSL, and DRNet—using a series of diagnostic tests.

The architectural framework of self-supervised HPE, where an encoder splits input images into separate appearance and pose codes for decoding and regression.

One primary test involved Latent Swapping, where the pose code of one subject is combined with the appearance code of another to synthesize a new image. If the codes are truly disentangled, the resulting image should show the second person in the first person's pose. The authors also utilized t-SNE visualizations to see if appearance codes clustered by person (as they should) or by action (indicating pose leakage), and developed a novel Appearance-only Adversarial Attack to see if modifying only the latent appearance vector would erroneously shift the predicted 3D joint positions.

Key Insights: The Disentanglement Failure

The most striking finding was that the pose code often contains significant appearance information. In experiments where the appearance code was replaced with a fixed zero vector, the decoder could still reconstruct specific details like shirt color or body shape, proving that the "pose channel" was contaminated.

Synthesized results from DRNet and CSSL showing the effects of swapping pose and appearance codes between different subjects (S7 and S8).

Because the pose code is contaminated, the supervised pose regressor learns to "cheat" by relying on appearance features rather than pure geometry. When the authors applied their adversarial attack to subtly change a subject's look in the latent space, the predicted 3D skeleton often collapsed or flipped, even though the physical pose remained identical. While data augmentation techniques like RGB jittering help slightly, they do not solve the underlying entanglement issue.

Results and Performance

Quantitatively, the failure of disentanglement is stark. In adversarial tests, the Mean Per-Joint Position Error (MPJPE) skyrocketed from a baseline of ~21.8 mm to over 175 mm. Using Distance Correlation (DC) metrics, the authors found high values (up to 0.88), where a value of 0 would indicate perfect independence. Among the tested models, DRNet was found to be the most "entangled," making it the most susceptible to visual changes.

Conclusion and Future Directions

This research exposes a critical vulnerability in current self-supervised 3D HPE frameworks. The reliance on contaminated pose codes means these models may not generalize well to new environments or diverse populations. Future work must focus on developing more sophisticated loss functions that explicitly penalize the leakage of appearance data into the pose channel and exploring advanced data augmentation strategies to force true geometric learning. The authors propose their adversarial framework as a necessary new benchmark for evaluating the robustness of future HPE systems.

Summary updated

Abstract

As 3D human pose estimation can now be achieved with very high accuracy in the supervised learning scenario, tackling the case where 3D pose annotations are not available has received increasing attention. In particular, several methods have proposed to learn image representations in a self-supervised fashion so as to disentangle the appearance information from the pose one. The methods then only need a small amount of supervised data to train a pose regressor using the pose-related latent vector as input, as it should be free of appearance information. In this paper, we carry out in-depth analysis to understand to what degree the state-of-the-art disentangled representation learning methods truly separate the appearance information from the pose one. First, we study disentanglement from the perspective of the self-supervised network, via diverse image synthesis experiments. Second, we investigate disentanglement with respect to the 3D pose regressor following an adversarial attack perspective. Specifically, we design an adversarial strategy focusing on generating natural appearance changes of the subject, and against which we could expect a disentangled network to be robust. Altogether, our analyses show that disentanglement in the three state-of-the-art disentangled representation learning frameworks if far from complete, and that their pose codes contain significant appearance information. We believe that our approach provides a valuable testbed to evaluate the degree of disentanglement of pose from appearance in self-supervised 3D human pose estimation.
BibTeX
@misc{nakka2023understanding,
  author = {Krishna Kanth Nakka and Mathieu Salzmann},
  title = {{Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation}},
  year = {2023},
  eprint = {2309.11667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.11667},
}
Figure