Learning Causal Representation for Training Cross-Domain Pose Estimator via Generative Interventions

Xiheng Zhang, Yongkang Wong, Xiaofei Wu, Juwei Lu, Mohan S. Kankanhalli, Xiangdong Li, Weidong Geng

3D human pose estimation has seen significant progress, yet a persistent challenge remains: models trained on controlled laboratory datasets often fail when deployed in diverse, real-world environments. This performance drop is primarily due to the "domain gap"—the difference in lighting, background, and subject appearance between training and testing data. In this paper, presented at ICCV 2021, researchers propose a novel framework that leverages causal inference and generative modeling to build a more robust, cross-domain estimator.

Overview of the proposed generative intervention framework for cross-domain 3D pose estimation.

The Challenge of Shortcut Learning

Deep learning models are notorious for being "shortcut learners." In the context of 3D pose estimation, a model might inadvertently learn that a specific background or a certain type of clothing is statistically associated with a particular joint configuration. These are known as spurious correlations.

When transitioning from a source domain (e.g., a laboratory) to a target domain (e.g., outdoor footage), these correlations break. Because the model relied on these shortcuts rather than the actual physical structure of the human body, its accuracy plummets. To achieve true generalization, the model must learn causal representations—features that represent the underlying physical pose and remain invariant across different environments.

A Causal Framework for Pose Estimation

The authors frame the image generation process through a Structural Causal Model (SCM). In this view, an image ($X$) is the result of two primary factors: Content ($C$), which represents the physical pose, and Domain ($D$), which encompasses environmental factors like lighting and background.

Causal graph representing the generation of images from content and domain factors.

As shown in the causal graph, the ground-truth pose ($Y$) is determined solely by the Content ($C$). However, the image features ($F$) extracted by a standard model are influenced by both $C$ and $D$. The goal of this research is to perform a "generative intervention" to break the link between the Domain and the extracted features, forcing the model to focus exclusively on the causal relationship between Content and Pose.

Generative Interventions via Counterfactuals

To achieve this, the researchers introduce a two-step generative process:

  1. Counterfactual Feature Generation: Instead of standard data augmentation, the team uses a specialized generator. This generator takes a known pose and random noise to produce latent features that mimic the statistical distribution of various domains. This is considered "counterfactual" because it creates a representation of a pose in an environment that does not exist in the original training set.
  2. Feature-Level Intervention: Unlike prior GAN-based methods that attempt to generate entire images (which often introduces artifacts like "ghosting"), this framework operates directly on the feature space. This ensures that the high-precision geometric information required for pose estimation is preserved.

Training Objectives

The pose estimator is trained using a multi-task objective:

  • Source Accuracy: Minimizing error on the original labeled source dataset.
  • Counterfactual Accuracy: Ensuring the model correctly predicts poses from the generated counterfactual features.
  • Distribution Matching: Utilizing Maximum Mean Discrepancy (MMD) to ensure the model's internal representations for real and counterfactual data are indistinguishable, effectively stripping away domain-specific noise.

Experimental Results and Impact

The method was rigorously evaluated across several benchmarks for both human and hand pose estimation, including Human3.6M, 3DPW, and FreiHAND. The results demonstrate state-of-the-art performance in both Domain Adaptation (where unlabeled target data is available) and Domain Generalization (where the target domain is entirely unknown).

  • Human Pose: On the challenging 3DPW dataset, the method improved the Procrustes Aligned Mean Per Joint Position Error (PA-MPJPE) by 6.9mm over existing baselines.
  • Hand Pose: The model significantly outperformed competitors across four distinct test datasets, showing high robustness against variations in skin color and complex backgrounds.
  • Generalization: By using diverse "unconstrained" datasets (like ImageNet or Kinetics) as intervention domains, the authors proved that the model could generalize to environments it had never encountered during training.

Conclusion

By shifting the focus from simple pattern matching to causal representation learning, this work provides a robust path forward for cross-domain computer vision. The "generative intervention" framework demonstrates that simulating the diversity of the real world in the latent feature space allows models to ignore nuisance variables and focus on the universal physical truths of human motion.

Summary updated

Abstract

3D pose estimation has attracted increasing attention with the availability of high-quality benchmark datasets. However, prior works show that deep learning models tend to learn spurious correlations, which fail to generalize beyond the specific dataset they are trained on. In this work, we take a step towards training robust models for cross-domain pose estimation task, which brings together ideas from causal representation learning and generative adversarial networks. Specifically, this paper introduces a novel framework for causal representation learning which explicitly exploits the causal structure of the task. We consider changing domain as interventions on images under the data-generation process and steer the generative model to produce counterfactual features. This help the model learn transferable and causal relations across different domains. Our framework is able to learn with various types of unlabeled datasets. We demonstrate the efficacy of our proposed method on both human and hand pose estimation task. The experiment results show the proposed approach achieves state-of-the-art performance on most datasets for both domain adaptation and domain generalization settings.
BibTeX
@inproceedings{zhang2021learning,
  author = {Xiheng Zhang and Yongkang Wong and Xiaofei Wu and Juwei Lu and Mohan S. Kankanhalli and Xiangdong Li and Weidong Geng},
  title = {{Learning Causal Representation for Training Cross-Domain Pose Estimator via Generative Interventions}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01108},
}
Figure