User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks

Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, Zhongxuan Luo

Abstract

Scribble colors based line art colorization is a challenging computer vision problem since neither greyscale values nor semantic information is presented in line arts, and the lack of authentic illustration-line art training pairs also increases difficulty of model generalization. Recently, several Generative Adversarial Nets (GANs) based methods have achieved great success. They can generate colorized illustrations conditioned on given line art and color hints. However, these methods fail to capture the authentic illustration distributions and are hence perceptually unsatisfying in the sense that they often lack accurate shading. To address these challenges, we propose a novel deep conditional adversarial architecture for scribble based anime line art colorization. Specifically, we integrate the conditional framework with WGAN-GP criteria as well as the perceptual loss to enable us to robustly train a deep network that makes the synthesized images more natural and real. We also introduce a local features network that is independent of synthetic data. With GANs conditioned on features from such network, we notably increase the generalization capability over "in the wild" line arts. Furthermore, we collect two datasets that provide high-quality colorful illustrations and authentic line arts for training and benchmarking. With the proposed model trained on our illustration dataset, we demonstrate that images synthesized by the presented approach are considerably more realistic and precise than alternative approaches.

Introduction

Colorizing anime line art is a labor-intensive process in the animation and illustration industries. While automated colorization for grayscale photos has seen great success, line art is significantly harder because it lacks the "hints" provided by shading and intensity values found in grayscale images. Previous deep learning approaches often suffered from two main issues: the "Generalization Gap" and a lack of realism. Most models are trained on synthetic line art generated by edge-detection filters. When these models encounter "in the wild" hand-drawn sketches by real artists, they often produce artifacts or unrealistic colors because the line characteristics differ. Furthermore, many GAN-based colorizers fail to capture authentic shading and lighting distributions, leading to flat or perceptually unsatisfying results.

Proposed Architecture

The proposed system utilizes a specialized conditional Generative Adversarial Network (cGAN) architecture designed to bridge the gap between synthetic training data and authentic artist sketches. A core innovation is the integration of a Local Features Network ($F_1$). Instead of feeding raw line art directly into the discriminator, the model uses a pretrained Illustration2Vec network to extract semantic and spatial feature maps. This network, trained on over a million illustrations, recognizes over 1,500 different tags (e.g., "blue hair," "uniform"), helping the model understand the semantic context of the image regardless of line style.

Overview of the proposed training pipeline featuring the local features network ($F_1$) and the conditional GAN framework.

The generator is a U-Net variant incorporating ResNeXt blocks to increase capacity and dilated convolutions to expand the receptive field without increasing computational overhead. To ensure realistic textures and stable training, the framework combines Wasserstein GAN with Gradient Penalty (WGAN-GP) and Perceptual Loss. The latter uses a pretrained VGG16 network to ensure the generated image's high-level features match the ground truth. During training, user interaction is simulated by randomly sampling pixels from the original colored illustration to serve as sparse color hints.

Experimental Results

The model was evaluated using two new high-quality benchmarks: an Illustration Dataset (21,930 images) and an Authentic Line Art Dataset (2,779 hand-drawn sketches). In quantitative tests, the proposed method achieved a Fréchet Inception Distance (FID) of 57.06, significantly outperforming the industry-standard PaintsChainer (FID 81.91). Qualitative human evaluation via Mean Opinion Score (MOS) also favored the proposed model, which scored 3.186 compared to PaintsChainer's range of 2.36 to 2.90.

Qualitative comparison of automatic and interactive colorization results against baseline models (Tanpopo, Satsuki, Canna).

Visually, the model demonstrates a superior ability to "hallucinate" complex shading, eye highlights, and gradients that respect the boundaries of the line art. By decoupling features from the training data through the $F_1$ network, the model avoids overfitting to synthetic lines, resulting in much cleaner performance on real-world artist sketches.

Conclusion and Future Work

This research presents a robust framework for user-guided line art colorization that significantly improves generalization to authentic hand-drawn art. While the results are state-of-the-art, some limitations remain, such as occasional failures in background color propagation and artifacts when user input is extremely sparse. Future work will focus on refining the interaction between global and local features to improve color consistency in complex scenes.

Summary updated

Abstract

Scribble colors based line art colorization is a challenging computer vision problem since neither greyscale values nor semantic information is presented in line arts, and the lack of authentic illustration-line art training pairs also increases difficulty of model generalization. Recently, several Generative Adversarial Nets (GANs) based methods have achieved great success. They can generate colorized illustrations conditioned on given line art and color hints. However, these methods fail to capture the authentic illustration distributions and are hence perceptually unsatisfying in the sense that they often lack accurate shading. To address these challenges, we propose a novel deep conditional adversarial architecture for scribble based anime line art colorization. Specifically, we integrate the conditional framework with WGAN-GP criteria as well as the perceptual loss to enable us to robustly train a deep network that makes the synthesized images more natural and real. We also introduce a local features network that is independent of synthetic data. With GANs conditioned on features from such network, we notably increase the generalization capability over "in the wild" line arts. Furthermore, we collect two datasets that provide high-quality colorful illustrations and authentic line arts for training and benchmarking. With the proposed model trained on our illustration dataset, we demonstrate that images synthesized by the presented approach are considerably more realistic and precise than alternative approaches.
BibTeX
@inproceedings{ci2018user,
  author = {Yuanzheng Ci and Xinzhu Ma and Zhihui Wang and Haojie Li and Zhongxuan Luo},
  title = {{User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks}},
  booktitle = {ACM Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=zWvzhSwqP8},
}
Figure