Scenimefy: Learning to Craft Anime Scene via Semi-Supervised Image-to-Image Translation

Yuxin Jiang, Liming Jiang, Shuai Yang, Chen Change Loy

TL;DR

Scenimefy is a novel semi-supervised image-to-image translation framework that automatically transforms real-world landscape photos into high-quality anime scenes. By utilizing a semantic-constrained StyleGAN to generate structurally consistent "pseudo-paired" data and introducing a patch-wise contrastive style loss, the method effectively captures intricate, hand-drawn anime textures while maintaining the semantic layout of complex scenes.

What Problem Does This Paper Address?

While automatic AI stylization has made massive strides in human faces, translating complex real-world scenes into high-quality anime style remains a formidable challenge. The authors identify three primary roadblocks:

  1. Scene Complexity: Natural landscapes contain complex spatial hierarchies and diverse objects (trees, clouds, mountains) with intricate relationships, unlike the relatively rigid structure of human faces.
  2. Unique Artistic Textures: Anime style isn't just about flat color blocks and sharp edges; it relies on complex, hand-drawn brush strokes for elements like grass and foliage.
  3. Data Scarcity: There is a severe lack of high-quality paired datasets (real vs. anime scenes). Existing unpaired datasets are often polluted with foreground human portraits, which degrades background scene synthesis.

Purely unsupervised image-to-image (I2I) translation methods often struggle to preserve semantics or capture fine texture details without pixel-wise guidance, leading to unnatural artifacts. Scenimefy bridges this gap by introducing a semi-supervised pipeline driven by high-quality pseudo-paired data.

Characteristics of anime scenes showing hand-drawn brush strokes.

Methods

Scenimefy splits the problem into a three-stage pipeline:

  1. Pseudo-Paired Data Generation: The authors fine-tune a source StyleGAN (pre-trained on real scenes) on an anime scene dataset. To ensure the generated anime images match the layout of the real images, they freeze the early, low-resolution layers of the generator. They also apply global CLIP and VGG perceptual losses alongside a patch-wise contrastive loss (PatchNCE) to enforce semantic consistency and retain local details during the domain shift.
  2. Semantic Segmentation-Guided Data Selection: Not all generated pseudo-pairs are perfect. The authors use a pre-trained segmentation model (Mask2Former) to generate semantic masks for both the real and synthesized anime images. Pairs that exhibit poor structural consistency (measured via cross-entropy loss) or lack semantic abundance (e.g., only containing a single object class) are automatically filtered out.
  3. Semi-Supervised I2I Translation: The core translation network features a dual-branch architecture. The unsupervised branch trains on the original unpaired datasets to capture the true target distribution. The supervised branch leverages the curated pseudo-paired data. Instead of using a strict $L_1$ reconstruction loss—which forces exact pixel matching and hurts stylization—they introduce a novel StylePatchNCE loss. This multi-scale patch-wise contrastive style loss maximizes mutual information between corresponding patches, training the model to mimic local anime textures without being penalized for minor spatial deviations.
Main framework of Scenimefy showing the dual-branch semi-supervised translation.

Datasets

The model was trained and evaluated using the following datasets:

  • Real Scenes: 90,000 natural landscape images from the Landscapes High-Quality (LHQ) dataset for training, and 6,656 scene images from the CycleGAN dataset for testing.
  • Anime Scenes (New Benchmark): The authors contributed a new high-resolution (1080x1080) dataset consisting of 5,958 pure anime scene images extracted from nine Makoto Shinkai films (e.g., Your Name, Weathering with You). The dataset was manually curated to remove human portraits and low-quality crops, narrowing the domain gap.
  • Validation Dataset: The Hosoda Mamoru dataset (5,107 images) was also used to demonstrate the framework's versatility across different anime art styles.
Examples from the newly introduced Shinkai-style anime scene dataset.

Key Insights and Ideas

  • The Value of "Soft" Supervision: Purely unsupervised learning is too unconstrained for complex scenes, but perfect paired data is impossible to collect. Providing "soft" supervision via structurally aligned pseudo-pairs strikes an ideal balance between style fidelity and semantic faithfulness.
  • Segmentation as a Quality Filter: Using cross-entropy between semantic segmentation masks of the source and generated domains acts as an effective automated gatekeeper, ensuring only high-quality structural pairs guide the network.
  • Texture over Pixels: Standard supervised translation relies heavily on pixel-level reconstruction losses, which often leads to blurry or overly conservative stylizations. Shifting to a patch-wise contrastive approach allows the network to learn the essence of the style (like brush strokes) rather than copying exact coordinates.

Key Results

Scenimefy significantly outperforms state-of-the-art baselines (including CartoonGAN, AnimeGAN, White-box, CTSS, and VToonify) both quantitatively and qualitatively:

  • Quantitative Metrics: Scenimefy achieved a Fréchet Inception Distance (FID) score of 48.922, substantially lower than the closest baseline (White-box at 61.973), indicating its generated distribution is much closer to real anime art.
  • User Study: In a perception study with 30 participants evaluating style, content preservation, and overall quality, Scenimefy secured the highest preference scores across all three criteria (e.g., an overall preference score of 0.673 vs. White-box's 0.103).
  • Qualitative Excellence: Visual results demonstrate that Scenimefy successfully renders complex hand-drawn textures (like individual leaves and clouds) while avoiding the unnatural artifacts and oversaturated color blocks common in prior methods.

Limitations and Future Work

Despite its success, Scenimefy has a few limitations:

  • Loss of Intricate Text Details: Because the model lacks rigid pixel-level constraints, it occasionally fails to preserve very fine details like text signs in background scenes.
  • Semantic Misinterpretations: In complex environments, the model can occasionally misinterpret objects (e.g., turning an ice mountain into a stone mountain).

For future work, the authors suggest exploring explicit controls for the degree of stylization and enabling user-input reference styles. They also note that integrating recent advancements in large-scale text-to-image diffusion models could yield even higher-quality pseudo-paired data to further elevate scene rendering.

Summary updated

Abstract

Automatic high-quality rendering of anime scenes from complex real-world images is of significant practical value. The challenges of this task lie in the complexity of the scenes, the unique features of anime style, and the lack of high-quality datasets to bridge the domain gap. Despite promising attempts, previous efforts are still incompetent in achieving satisfactory results with consistent semantic preservation, evident stylization, and fine details. In this study, we propose Scenimefy, a novel semi-supervised image-to-image translation framework that addresses these challenges. Our approach guides the learning with structure-consistent pseudo paired data, simplifying the pure unsupervised setting. The pseudo data are derived uniquely from a semantic-constrained StyleGAN leveraging rich model priors like CLIP. We further apply segmentation-guided data selection to obtain high-quality pseudo supervision. A patch-wise contrastive style loss is introduced to improve stylization and fine details. Besides, we contribute a high-resolution anime scene dataset to facilitate future research. Our extensive experiments demonstrate the superiority of our method over state-of-the-art baselines in terms of both perceptual quality and quantitative performance.
BibTeX
@inproceedings{jiang2023scenimefy,
  author = {Yuxin Jiang and Liming Jiang and Shuai Yang and Chen Change Loy},
  title = {{Scenimefy: Learning to Craft Anime Scene via Semi-Supervised Image-to-Image Translation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure