One-Shot Stylization for Full-Body Human Images

Aiyu Cui, Svetlana LazebnikarXiv 2023 · DOI · arXiv

Introduction

The proliferation of "cartoon filters" has made facial stylization a common feature in social media. However, extending this capability to full-body human images remains a significant technical hurdle. Unlike facial images, which are relatively constrained in geometry, full-body images involve a vast array of poses, complex clothing textures, and diverse backgrounds. This paper presents a novel framework for one-shot full-body stylization, enabling the transformation of a real-world photo into the aesthetic of a single artistic reference image while preserving the subject's pose and identity.

The Challenge of Full-Body Stylization

The authors identify two primary gaps in existing style transfer methods when applied to the human form:

  1. Structural Complexity and Entanglement: Full-body images feature intricate details like fabric folds and limb positions. Standard generative models often "tangle" pose and appearance, meaning that changing the style (e.g., to a sketch or anime) often results in a loss of the original pose or clothing structure.
  2. Non-Human Body Proportions: Artistic styles frequently rely on exaggerated geometry. For instance, characters in films like Frozen or Coraline possess head-to-body ratios and limb lengths that differ drastically from realistic human anatomy. Traditional style transfer focuses on color and texture (the "look"), leaving the realistic skeleton intact and creating an "uncanny valley" effect where the character looks like a human wearing a mask rather than a cohesive artistic creation.

Proposed Methodology

To address these challenges, the researchers developed a three-stage pipeline centered on a pose-guided generator.

The "Destylization" Trick

A major hurdle in training stylization models is the lack of paired data (i.e., a photo of a person and a corresponding drawing of that same person in the same pose). To circumvent this, the authors propose a "destylization" step. They use a pre-trained human generator ($G_0$) to project the artistic reference image into a photo-realistic space. This creates a synthetic "photo" version of the artistic character, providing a pseudo-paired training example.

Joint Fine-tuning and Warping

With the synthetic pair in hand, a generator ($G_a$) is fine-tuned to map the destylized photo back to the original artistic style. To prevent the model from overfitting to the single reference image, the authors employ a "joint task" strategy: the model must simultaneously learn to stylize a diverse dataset of real human photos using standard perceptual and style losses.

The base architecture is an enhanced version of the "Dressing in Order" (DiOr) model, dubbed DiOr+. The authors found that warping-based generators, which physically shift pixels, are superior to encoder-decoder models for preserving complex clothing patterns during this process.

Learnable Skeleton Deformation

Perhaps the most significant innovation is the skeleton deformation module. The system uses a Multi-Layer Perceptron (MLP) to predict "body ratio factors" based on the artistic reference. Before the final image generation, the input person's skeleton is rescaled to match the proportions of the target style. This allows the model to generate characters with the large heads or elongated limbs characteristic of specific art styles.

The framework demonstrates the ability to transfer diverse artistic styles onto various human poses and outfits while adjusting body proportions.

Results and Evaluation

The method was tested against a gallery of 26 artistic references, including anime (K-On!), films (Frozen), and classical art (Modigliani).

Qualitative and Quantitative Success

The system successfully handles "one-shot" cases, maintaining the subject's specific clothing and pose while convincingly applying the character's aesthetic and proportions. In a user preference study, the proposed method was preferred over NNST (Neural Neighbor Style Transfer) 76.1% of the time. Quantitatively, the model achieved significantly lower LPIPS scores on art reconstruction tasks compared to other state-of-the-art models like ADGAN or NTED, indicating a higher fidelity to the target style.

An example of the model's ability to capture exaggerated body proportions, such as the enlarged head of a cartoon style, while maintaining clothing details.

Conclusion and Limitations

While the framework represents a significant step forward, it is not without limitations. The output quality is inherently tied to the capabilities of the base generator; if the base model struggles with rendering hands or overlapping limbs, those artifacts will persist. Additionally, the "destylization" step can struggle with highly abstract or grayscale art, occasionally failing to learn how to "decolorization" a photo. Future work may focus on integrating more robust base models and improving the handling of non-color-based artistic elements.

Summary updated

Abstract

The goal of human stylization is to transfer full-body human photos to a style specified by a single art character reference image. Although previous work has succeeded in example-based stylization of faces and generic scenes, full-body human stylization is a more complex domain. This work addresses several unique challenges of stylizing full-body human images. We propose a method for one-shot fine-tuning of a pose-guided human generator to preserve the "content" (garments, face, hair, pose) of the input photo and the "style" of the artistic reference. Since body shape deformation is an essential component of an art character's style, we incorporate a novel skeleton deformation module to reshape the pose of the input person and modify the DiOr pose-guided person generator to be more robust to the rescaled poses falling outside the distribution of the realistic poses that the generator is originally trained on. Several human studies verify the effectiveness of our approach.
BibTeX
@misc{cui2023one,
  author = {Aiyu Cui and Svetlana Lazebnik},
  title = {{One-Shot Stylization for Full-Body Human Images}},
  year = {2023},
  eprint = {2304.06917},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.06917},
}
Figure