Abstract
Recently, customized generation has shown significant potential by using as few as 3-5 user-provided images to train a model to synthesize new images of a specified subject. While subsequent applications have enhanced the flexibility and diversity of customized generation, fine-grained control over a specified subject mimicking a human's pose remains under-studied. In this paper, we propose PersonificationNet, a framework designed to control non-human subjects—such as cartoon characters or plush toys—to perform the same pose as a given reference person. The system consists of a customized branch for appearance, a pose condition branch for structure transfer, and a structure alignment module to bridge the anatomical gap between humans and variant instances. Experimental results demonstrate that PersonificationNet outperforms state-of-the-art methods in generating anatomically coherent and pose-accurate images.
Introduction
Recent breakthroughs in text-to-image models have popularized "customized generation" (e.g., DreamBooth), allowing users to teach a model a specific subject using just a handful of images. However, getting these subjects to perform specific actions—like a reference human pose—remains a major challenge due to the inherent differences in body proportions between humans and non-human subjects.
The authors identify two main hurdles in existing workflows:
- The Structure Gap: Existing pose-control tools like ControlNet are trained on human proportions. When applying a human pose to a non-human subject (like a round potato or a stubby sloth), models struggle to reconcile the human-like skeleton with the subject's actual body shape, leading to distorted or "uncanny" results.
- Prior Knowledge Bias: Standard pose branches inherently rely on human body priors, making them ineffective at controlling variant body structures.
Methodology
PersonificationNet introduces a three-part architecture to solve these issues:
Customized Branch
This follows the DreamBooth approach. It uses a small set of user-provided images and a unique identifier (e.g., "sks Mr. Potato Head") to inject the subject's appearance into a Stable Diffusion model, ensuring high fidelity to the original subject.
Pose Condition Branch
Instead of using a "frozen" ControlNet, the authors fine-tune the pose branch on a specialized dataset of diverse body structures. This teaches the model to be more flexible and less biased toward strictly human proportions, allowing it to better interpret how non-humanoid skeletons should move.
Structure Alignment Module (SAM)
This is the core innovation used during the inference stage. It bridges the gap between the human reference and the target subject through a four-step process:
- Node Identification: Identifies "father nodes" (like shoulder or hip centers).
- Coordinate Transformation: Converts skeleton joint coordinates from Euclidean $(x, y)$ to polar coordinates $(\rho, \theta)$.
- Angle Mapping: Maps the angles $(\theta)$ from the human reference pose onto the limb lengths $(\rho)$ of the target subject.
- Reconstruction: Converts back to Euclidean space to create a "subject-proportioned" pose image for the model to follow.
Experimental Results
The paper primarily showcases qualitative improvements over state-of-the-art (SOTA) baselines like DreamBooth + ControlNet. By decoupling action from structure using polar coordinates, the model preserves the action of the human reference while respecting the proportions of the subject.
Visual Coherence and Ablation
While baseline methods often produce "human-limbs-on-subject" artifacts (e.g., Mr. Potato Head having long, fleshy human legs), PersonificationNet generates images where the subject's own anatomy performs the action. Ablation studies demonstrate that without the Structure Alignment Module, the subject often reverts to human proportions. Without the fine-tuned pose branch, the model produces significant artifacts like floating limbs or inconsistent textures.
Conclusion and Future Work
PersonificationNet bridges the "structure gap" in customized image generation, allowing non-human subjects to naturally mimic human poses. While the paper demonstrates a significant leap, future work includes automating skeleton annotation for non-humanoid shapes and testing on more extreme morphologies. Scaling the fine-tuning dataset beyond the current 55 images could further improve robustness across a wider variety of abstract subjects.
Summary updated