Abstract
The transformation of sparse line sketches into fully colored illustrations is a challenging, ill-posed problem in computer vision. This paper introduces Auto-painter, a framework leveraging Conditional Generative Adversarial Networks (cGANs) to automate this process. By utilizing a U-Net generator with skip connections and a specialized loss function—comprising adversarial, pixel-level, perceptual, and smoothness constraints—the model produces vibrant, high-quality cartoon images. Experimental results demonstrate that Auto-painter significantly outperforms baseline models in visual quality and user preference, while also providing an interactive mechanism for user-guided color control.
Introduction
Creating digital art from a simple sketch is a labor-intensive task that requires significant artistic skill. Unlike grayscale-to-color conversion, where textures provide hints, sketches contain minimal information, requiring a model to "imagine" textures, lighting, and non-realistic color schemes common in animation. Previous generative approaches often resulted in blurry outputs or color bleeding. Auto-painter addresses these issues by providing a tool that assists both non-artists and professionals in rapid prototyping and high-quality image generation.
Methodology
The Auto-painter model is built upon the cGAN framework, optimized for the specific constraints of sketch colorization.
Architecture
The generator employs a U-Net architecture. Unlike standard encoder-decoders, U-Net uses skip connections to pass low-level edge information directly to the output layers, ensuring the final colorization adheres strictly to the input sketch's lines. The discriminator is a PatchGAN, which evaluates the image at the scale of 70x70 local patches. This focus on high-frequency details helps maintain crisp edges and realistic local textures.
Loss Function
To stabilize training and improve visual fidelity, the authors implemented a composite loss function:
- Adversarial Loss ($L_G$): Drives the generation of vivid, realistic images.
- Pixel-level $L_1$ Loss ($L_p$): Ensures spatial alignment with the ground truth.
- Feature Loss ($L_f$): Uses a pre-trained VGG-16 network to compare high-level semantic features, preventing blurriness.
- Total Variation Loss ($L_{tv}$): Acts as a smoothness constraint to reduce noise and color bleeding.
Datasets and Training
The model was trained on two datasets:
- Minions: 1,100 images of Minion characters.
- Japanimation: 60,000 images of Japanese anime characters. To create paired training data, the authors used the XDoG filter to synthetically generate sketches from colored images, simulating various levels of artistic detail.
Results and Discussion
Subjective testing with 55 volunteers compared Auto-painter against the Pix2Pix baseline. Auto-painter achieved a popularity score of 1.948, significantly higher than Pix2Pix's -1.524. The model demonstrated high reliability and low variance across diverse sketch types.
A key feature of Auto-painter is its interactive control. By training with "color blocks" (randomly sampled pixels from the ground truth), the model allows users to provide color hints. The network then propagates these hints naturally within object boundaries, as seen in the comparison between standard auto-painting and guided painting (Figure 3).
Conclusion
Auto-painter provides an efficient solution for sketch colorization, generating 512x512 images in under one second. While the model is sensitive to hyperparameter tuning and requires significant training time, it represents a substantial step toward automated digital art tools. Future work will focus on further optimizing performance and developing a comprehensive interactive toolkit for artists.
Summary updated