User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks

Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, Zhongxuan Luo

TL;DR

This paper introduces a novel conditional generative adversarial network (cGAN) designed to colorize anime line art based on sparse user "scribbles." By integrating a pretrained local features network and utilizing Wasserstein GAN (WGAN-GP) objectives, the model overcomes the common "generalization gap" where AI trained on synthetic sketches fails to perform on authentic, hand-drawn art.

What Problem Does This Paper Address?

Colorizing anime line art is a labor-intensive process in the animation and illustration industries. While automated colorization for grayscale photos has seen great success, line art is significantly harder because it lacks the "hints" provided by shading and intensity values found in grayscale images.

Previous deep learning approaches often suffered from two main issues:

  1. The Generalization Gap: Most models are trained on synthetic line art (generated by edge-detection filters like XDoG). When these models encounter "in the wild" hand-drawn sketches by real artists, they often produce artifacts or unrealistic colors because the line characteristics differ.
  2. Lack of Realism: Many GAN-based colorizers fail to capture authentic shading and lighting distributions, leading to flat or perceptually unsatisfying results.

Methods

The authors propose a specialized cGAN architecture with several key technical components:

  • Local Features Network ($F_1$): Instead of feeding raw line art directly into the discriminator, the authors use a pretrained Illustration2Vec network to extract semantic and spatial feature maps. This network is trained on over a million illustrations to recognize 1,539 different tags (e.g., "blue hair," "uniform"). This helps the model understand what it is colorizing, regardless of whether the lines are synthetic or hand-drawn.
  • Advanced Generator Architecture: The generator is a U-Net variant that incorporates ResNeXt blocks to increase network capacity and dilated convolutions to expand the receptive field without increasing computational cost. It also uses sub-pixel convolution layers for high-quality upsampling.
  • Robust Loss Functions: To ensure realistic textures and stable training, the team moved away from "vanilla" GAN loss. They combined:
  • WGAN-GP: For more stable training and better convergence.
  • Perceptual Loss: Using a pretrained VGG16 network to ensure the generated image's high-level features match the ground truth.
  • Simulated User Interaction: During training, the model "learns" from sparse color hints by randomly sampling pixels from the original colored illustration and feeding them in as a "hint" tensor.

Datasets

The authors noted that existing datasets were often "messy" (containing scribbles or low-quality sketches). They introduced two new high-quality benchmarks:

  1. Illustration Dataset: 21,930 high-quality colored anime illustrations used for training.
  2. Authentic Line Art Dataset: 2,779 hand-drawn line arts collected from the internet, used specifically for testing "in the wild" performance.

Key Insights and Ideas

  • Decoupling Features from Training: By using a local features network ($F_1$) that is independent of the synthetic training data, the model avoids overfitting to the specific "look" of computer-generated lines. This is the "secret sauce" for why it works so well on real artist sketches.
  • No Normalization: Interestingly, the authors chose not to use any normalization layers (like Batch Norm). They found this preserved "range flexibility," allowing for more accurate and vibrant color reproduction.
  • Conditioning the Discriminator: The discriminator is also conditioned on the local features from $F_1$. This forces the discriminator to judge not just if an image looks "real," but if the colors make semantic sense for the specific objects in the line art.

Key Results

The model was compared against PaintsChainer, a popular industry-standard tool.

  • Quantitative (FID): The proposed method achieved a Fréchet Inception Distance (FID) of 57.06, significantly lower (better) than PaintsChainer’s best model (81.91). A lower FID indicates the generated images are statistically much closer to real human-drawn illustrations.
  • Qualitative (MOS): In a Mean Opinion Score test with 16 human raters, the model scored 3.186, outperforming all versions of PaintsChainer (which scored between 2.36 and 2.90).
  • Visual Performance: The model shows a superior ability to "hallucinate" complex shading, eye highlights, and gradients that respect the boundaries of the line art.

Limitations and Future Work

While the results are impressive, the authors noted some failure cases:

  • Background Propagation: In some instances, background colors provided by the user do not propagate smoothly across the entire image (see Figure 4, row 2).
  • Sparse Input Artifacts: When user input is extremely sparse, the automatic results can sometimes produce "gridding artifacts" or inconsistent textures.
  • Future Directions: The authors suggest that further refining the interaction between global and local features could improve the consistency of color propagation in complex scenes.

Summary updated

Abstract

Scribble colors based line art colorization is a challenging computer vision problem since neither greyscale values nor semantic information is presented in line arts, and the lack of authentic illustration-line art training pairs also increases difficulty of model generalization. Recently, several Generative Adversarial Nets (GANs) based methods have achieved great success. They can generate colorized illustrations conditioned on given line art and color hints. However, these methods fail to capture the authentic illustration distributions and are hence perceptually unsatisfying in the sense that they often lack accurate shading. To address these challenges, we propose a novel deep conditional adversarial architecture for scribble based anime line art colorization. Specifically, we integrate the conditional framework with WGAN-GP criteria as well as the perceptual loss to enable us to robustly train a deep network that makes the synthesized images more natural and real. We also introduce a local features network that is independent of synthetic data. With GANs conditioned on features from such network, we notably increase the generalization capability over "in the wild" line arts. Furthermore, we collect two datasets that provide high-quality colorful illustrations and authentic line arts for training and benchmarking. With the proposed model trained on our illustration dataset, we demonstrate that images synthesized by the presented approach are considerably more realistic and precise than alternative approaches.
BibTeX
@inproceedings{ci2018user,
  author = {Yuanzheng Ci and Xinzhu Ma and Zhihui Wang and Haojie Li and Zhongxuan Luo},
  title = {{User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks}},
  booktitle = {ACM Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=zWvzhSwqP8},
}

Loading related content...

Figure