FlexPose: Pose Distribution Adaptation with Limited Guidance

Z. Jane Wang, Junwu Weng, Mengyuan Liu, Bei Yu

Abstract

Numerous well-annotated human key-point datasets are publicly available to date. However, annotating human poses for newly collected images is still a costly and time-consuming progress. Pose distributions from different datasets share similar pose hinge-structure priors with different geometric transformations, such as pivot orientation, joint rotation, and bone length ratio. The difference between Pose distributions is essentially the difference between the transformation distributions. Inspired by this fact, we propose a method to calibrate a pre-trained pose generator in which the pose prior has already been learned to an adapted one following a new pose distribution. We treat the representation of human pose joint coordinates as skeleton image and transfer a pre-trained pose annotation generator with only a few annotation guidance. By fine-tuning a limited number of linear layers that closely related to the pose transformation, the adapted generator is able to produce any number of pose annotations that are similar to the target poses. We evaluate our proposed method, FlexPose, on several cross-dataset settings both qualitatively and quantitatively, which demonstrates that our approach achieves state-of-the-art performance compared to the existing generative-model-based transfer learning methods when given limited annotation guidance.

Introduction

Deep learning models for human pose estimation require massive amounts of annotated data. While large public datasets like COCO exist, applying models trained on these to new domains often fails due to domain shift. For instance, a model trained on "in-the-wild" photos might struggle with studio-captured medical data because camera angles, bone length ratios, and typical activities differ. Traditional domain adaptation often relies on raw images, raising privacy concerns and demographic biases. FlexPose addresses this by focusing purely on the pose distribution, leveraging the universal "hinge-structure" of human anatomy while adapting geometric transformations like rotation and scale.

Methodology

FlexPose utilizes a three-phase approach to adapt pose distributions with minimal guidance.

The three-phase FlexPose pipeline: Generic Pose Distribution Estimation, Pose Distribution Adaptation, and Target Pose Sampling.
  1. Generic Pose Distribution Estimation: 2D keypoints are represented as "skeleton images"—colored stick figures where each bone has a unique color. A StyleGAN generator is trained to learn the anatomical prior from a large source dataset.
  2. Pose Distribution Adaptation: Instead of retraining the entire model, a lightweight, learnable linear module is inserted into the generator. The authors identified that coarse layers (specifically layer 3) control global geometry. By tuning only these layers, the model avoids overfitting to the sparse target examples. To expand the limited target set, "pose-mixup" is performed on keypoint coordinates to create physically plausible intermediate poses.
  3. Target Pose Sampling: The adapted generator can then synthesize infinite poses matching the target domain's style, which can be used to train downstream pose detectors.

Experiments and Results

The framework was evaluated across several cross-dataset scenarios, including COCO to Human3.6M and face landmark adaptation from 300-VW to WFLW.

t-SNE visualization showing the distribution of Human3.6M, COCO, and Adapted COCO poses.

FlexPose achieved superior distribution matching, reaching a Frechet Distance (FD) of 0.80 on the COCO to Human3.6M task, significantly outperforming methods like LoRA (1.36). Downstream accuracy also saw marked improvements; for the 3DHP to Human3.6M task, the Percentage of Correct Keypoints (PCK) jumped from 0.0% to 46.7%. Furthermore, the adaptation process is highly efficient, requiring less than 5 minutes on a single V100 GPU.

Discussion and Conclusion

The core innovation of FlexPose lies in its "geometric lever" approach—adjusting global pose orientation and scale through specific GAN layers rather than retraining the entire concept of a human pose. By using skeleton images, the method preserves spatial relationships and ensures privacy by avoiding raw RGB data. While highly effective, the method's performance is tied to the quality of the source prior and may require more guidance for extreme outlier poses in the target distribution. Future work includes extending this framework to 3D pose adaptation.

Summary updated

Abstract

Numerous well-annotated human key-point datasets are publicly available to date. However, annotating human poses for newly collected images is still a costly and time-consuming progress. Pose distributions from different datasets share similar pose hinge-structure priors with different geometric transformations, such as pivot orientation, joint rotation, and bone length ratio. The difference between Pose distributions is essentially the difference between the transformation distributions. Inspired by this fact, we propose a method to calibrate a pre-trained pose generator in which the pose prior has already been learned to an adapted one following a new pose distribution. We treat the representation of human pose joint coordinates as skeleton image and transfer a pre-trained pose annotation generator with only a few annotation guidance. By fine-tuning a limited number of linear layers that closely related to the pose transformation, the adapted generator is able to produce any number of pose annotations that are similar to the target poses. We evaluate our proposed method, FlexPose, on several cross-dataset settings both qualitatively and quantitatively, which demonstrates that our approach achieves state-of-the-art performance compared to the existing generative-model-based transfer learning methods when given limited annotation guidance.
BibTeX
@inproceedings{wang2025flexpose,
  author = {Z. Jane Wang and Junwu Weng and Mengyuan Liu and Bei Yu},
  title = {{FlexPose: Pose Distribution Adaptation with Limited Guidance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35446},
}
Figure