PersonificationNet: Making customized subject act like a person

Tianchu Guo, Pengyu Li, Biao Wang, Xian-Sheng HuaarXiv 2024 · OpenReview

Abstract

Recently, customized generation has shown significant potential by using as few as 3-5 user-provided images to train a model to synthesize new images of a specified subject. While subsequent applications have enhanced the flexibility and diversity of customized generation, fine-grained control over a specified subject mimicking a human's pose remains under-studied. In this paper, we propose PersonificationNet, a framework designed to control non-human subjects—such as cartoon characters or plush toys—to perform the same pose as a given reference person. The system consists of a customized branch for appearance, a pose condition branch for structure transfer, and a structure alignment module to bridge the anatomical gap between humans and variant instances. Experimental results demonstrate that PersonificationNet outperforms state-of-the-art methods in generating anatomically coherent and pose-accurate images.

Introduction

Recent breakthroughs in text-to-image models have popularized "customized generation" (e.g., DreamBooth), allowing users to teach a model a specific subject using just a handful of images. However, getting these subjects to perform specific actions—like a reference human pose—remains a major challenge due to the inherent differences in body proportions between humans and non-human subjects.

The task of PersonificationNet: transferring a human pose from a reference image to a customized subject.

The authors identify two main hurdles in existing workflows:

  1. The Structure Gap: Existing pose-control tools like ControlNet are trained on human proportions. When applying a human pose to a non-human subject (like a round potato or a stubby sloth), models struggle to reconcile the human-like skeleton with the subject's actual body shape, leading to distorted or "uncanny" results.
  2. Prior Knowledge Bias: Standard pose branches inherently rely on human body priors, making them ineffective at controlling variant body structures.

Methodology

PersonificationNet introduces a three-part architecture to solve these issues:

Customized Branch

This follows the DreamBooth approach. It uses a small set of user-provided images and a unique identifier (e.g., "sks Mr. Potato Head") to inject the subject's appearance into a Stable Diffusion model, ensuring high fidelity to the original subject.

Pose Condition Branch

Instead of using a "frozen" ControlNet, the authors fine-tune the pose branch on a specialized dataset of diverse body structures. This teaches the model to be more flexible and less biased toward strictly human proportions, allowing it to better interpret how non-humanoid skeletons should move.

Structure Alignment Module (SAM)

This is the core innovation used during the inference stage. It bridges the gap between the human reference and the target subject through a four-step process:

  1. Node Identification: Identifies "father nodes" (like shoulder or hip centers).
  2. Coordinate Transformation: Converts skeleton joint coordinates from Euclidean $(x, y)$ to polar coordinates $(\rho, \theta)$.
  3. Angle Mapping: Maps the angles $(\theta)$ from the human reference pose onto the limb lengths $(\rho)$ of the target subject.
  4. Reconstruction: Converts back to Euclidean space to create a "subject-proportioned" pose image for the model to follow.

Experimental Results

The paper primarily showcases qualitative improvements over state-of-the-art (SOTA) baselines like DreamBooth + ControlNet. By decoupling action from structure using polar coordinates, the model preserves the action of the human reference while respecting the proportions of the subject.

Comparison of PersonificationNet against baseline methods. Our approach avoids the 'structure gap' artifacts seen in DreamBooth and ControlNet combinations.

Visual Coherence and Ablation

While baseline methods often produce "human-limbs-on-subject" artifacts (e.g., Mr. Potato Head having long, fleshy human legs), PersonificationNet generates images where the subject's own anatomy performs the action. Ablation studies demonstrate that without the Structure Alignment Module, the subject often reverts to human proportions. Without the fine-tuned pose branch, the model produces significant artifacts like floating limbs or inconsistent textures.

Conclusion and Future Work

PersonificationNet bridges the "structure gap" in customized image generation, allowing non-human subjects to naturally mimic human poses. While the paper demonstrates a significant leap, future work includes automating skeleton annotation for non-humanoid shapes and testing on more extreme morphologies. Scaling the fine-tuning dataset beyond the current 55 images could further improve robustness across a wider variety of abstract subjects.

Summary updated

Abstract

Recently customized generation has significant potential, which uses as few as 3-5 user-provided images to train a model to synthesize new images of a specified subject. Though subsequent applications enhance the flexibility and diversity of customized generation, fine-grained control over the given subject acting like the person's pose is still lack of study. In this paper, we propose a PersonificationNet, which can control the specified subject such as a cartoon character or plush toy to act the same pose as a given referenced person's image. It contains a customized branch, a pose condition branch and a structure alignment module. Specifically, first, the customized branch mimics specified subject appearance. Second, the pose condition branch transfers the body structure information from the human to variant instances. Last, the structure alignment module bridges the structure gap between human and specified subject in the inference stage. Experimental results show our proposed PersonificationNet outperforms the state-of-the-art methods.
BibTeX
@misc{guo2024personificationnet,
  author = {Tianchu Guo and Pengyu Li and Biao Wang and Xian-Sheng Hua},
  title = {{PersonificationNet: Making customized subject act like a person}},
  year = {2024},
  eprint = {2407.09057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.09057},
}
Figure