Towards Fine-grained Human Pose Transfer with Detail Replenishing Network

Lingbo Yang, Pan Wang, Chang Liu, Zhanning Gao, Peiran Ren, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Xiansheng Hua, Wen Gao

Abstract

Human pose transfer (HPT) is an emerging research topic with huge potential in fashion design, media production, online advertising and virtual reality. For these applications, the visual realism of fine-grained appearance details is crucial for production quality and user engagement. However, existing HPT methods often suffer from three fundamental issues: detail deficiency, content ambiguity and style inconsistency, which severely degrade the visual quality and realism of generated images. Aiming towards real-world applications, we develop a more challenging yet practical HPT setting, termed as Fine-grained Human Pose Transfer (FHPT), with a higher focus on semantic fidelity and detail replenishment. Concretely, we analyze the potential design flaws of existing methods via an illustrative example, and establish the core FHPT methodology by combing the idea of content synthesis and feature transfer together in a mutually-guided fashion. Thereafter, we substantiate the proposed methodology with a Detail Replenishing Network (DRN) and a corresponding coarse-to-fine model training scheme. Moreover, we build up a complete suite of fine-grained evaluation protocols to address the challenges of FHPT in a comprehensive manner, including semantic analysis, structural detection and perceptual quality assessment. Extensive experiments on the DeepFashion benchmark dataset have verified the power of proposed benchmark against start-of-the-art works, with 12%-14% gain on top-10 retrieval recall, 5% higher joint localization accuracy, and near 40% gain on face identity preservation.

Introduction

Human Pose Transfer (HPT) involves synthesizing a new image of a person based on a source image and a target pose. Despite significant progress, current HPT methods struggle with three primary issues that hinder their use in high-fidelity applications like virtual reality or fashion design:

  1. Detail Deficiency: Global generation methods often produce over-smoothed results, losing crisp textures in clothing and faces.
  2. Content Ambiguity: Local warping methods fail when body parts occluded in the source image become visible in the target pose.
  3. Style Inconsistency: Hybrid models attempting to "hallucinate" missing regions often generate textures that do not match the original garment's style.

To address these, this work introduces Fine-grained Human Pose Transfer (FHPT), a setting that prioritizes semantic fidelity and the replenishment of fine details.

Methodology: Detail Replenishing Network (DRN)

The proposed Detail Replenishing Network (DRN) employs a coarse-to-fine strategy to bridge the gap between pose alignment and detail preservation. The architecture is built upon the realization that detail replenishment can be treated as a style-transfer problem guided by spatial information.

Evolution of HPT architectures addressing detail deficiency, content ambiguity, and style inconsistency leading towards FHPT.

The DRN consists of two main branches:

  • Pose Transfer Branch: Focuses on spatial alignment, generating a coarse estimation and a spatial guidance map using cascaded transfer blocks.
  • Detail Replenishing Branch: Extracts a global "style code" from the source image and uses Adaptive Instance Normalization (AdaIN) to infuse this style into the generation process, ensuring consistency even in hallucinated regions.
  • Regional Face Module: A dedicated module uses face detection and landmark sketches to specifically reconstruct facial identity, mitigating common issues like gender bias where models "feminize" male subjects due to dataset imbalances.

These branches operate with Mutual Guidance, where the coarse branch provides spatial hints and the detail branch provides the residual map necessary to sharpen the final output.

Experiments and Evaluation

The model was evaluated on the DeepFashion dataset. Moving beyond traditional metrics like SSIM, which can be "blind" to fine details, the authors introduced a suite of fine-grained protocols including Face Identity Preservation, Keypoint Error Curves (KEC), and Content-Based Image Retrieval (CBIR).

Qualitative comparison of DRN against state-of-the-art methods (DSC, PATN) and a summary of semantic fidelity improvements across various attributes.

Key Results

The DRN significantly outperforms state-of-the-art methods like DSC and PATN across nearly every metric:

  • Identity Preservation: Achieved a near 40% gain in face identity preservation.
  • Retrieval Accuracy: Showed a 12%-14% gain in top-10 retrieval recall, indicating superior semantic accuracy.
  • Structural Integrity: Improved joint localization accuracy by 5%.
  • Qualitative Quality: The DRN successfully recovers complex patterns (e.g., lace, leopard print) and maintains masculine features in male subjects where other models fail.

Conclusion

The Detail Replenishing Network establishes a new benchmark for fine-grained human pose transfer by treating detail replenishment as a style-guided synthesis task. By leveraging mutual guidance between spatial and style branches, the model ensures both structural accuracy and textural consistency. Future work will explore more granular, region-adaptive style control to handle outfits with diverse fabrics and improve generalization to complex real-world backgrounds.

Summary updated

Abstract

Human pose transfer (HPT) is an emerging research topic with huge potential in fashion design, media production, online advertising and virtual reality. For these applications, the visual realism of fine-grained appearance details is crucial for production quality and user engagement. However, existing HPT methods often suffer from three fundamental issues: detail deficiency, content ambiguity and style inconsistency, which severely degrade the visual quality and realism of generated images. Aiming towards real-world applications, we develop a more challenging yet practical HPT setting, termed as Fine-grained Human Pose Transfer (FHPT), with a higher focus on semantic fidelity and detail replenishment. Concretely, we analyze the potential design flaws of existing methods via an illustrative example, and establish the core FHPT methodology by combing the idea of content synthesis and feature transfer together in a mutually-guided fashion. Thereafter, we substantiate the proposed methodology with a Detail Replenishing Network (DRN) and a corresponding coarse-to-fine model training scheme. Moreover, we build up a complete suite of fine-grained evaluation protocols to address the challenges of FHPT in a comprehensive manner, including semantic analysis, structural detection and perceptual quality assessment. Extensive experiments on the DeepFashion benchmark dataset have verified the power of proposed benchmark against start-of-the-art works, with 12\%-14\% gain on top-10 retrieval recall, 5\% higher joint localization accuracy, and near 40\% gain on face identity preservation. Moreover, the evaluation results offer further insights to the subject matter, which could inspire many promising future works along this direction.
BibTeX
@article{yang2020towards,
  author = {Lingbo Yang and Pan Wang and Chang Liu and Zhanning Gao and Peiran Ren and Xinfeng Zhang and Shanshe Wang and Siwei Ma and Xiansheng Hua and Wen Gao},
  title = {{Towards Fine-grained Human Pose Transfer with Detail Replenishing Network}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  doi = {10.1109/tip.2021.3052364},
}
Figure