TL;DR
This paper introduces "late-resizing," a novel strategy for sketch extraction that significantly improves the generalization of line-art colorization models. By performing resizing after line detection, the method mitigates domain gaps between synthetic and real sketches, leading to more robust and higher-quality colorization.
What Problem Does This Paper Address?
Automatic line-art colorization, the process of adding color to black-and-white line drawings, is a challenging task. While deep learning has shown promise, it requires large datasets of paired line art and color images. Creating such datasets is labor-intensive. To overcome this, researchers often use "sketch extraction" to automatically generate line art from existing color images. However, a critical step in this process, resizing the input images, can introduce artifacts and create a "domain gap" between the synthetically generated sketches and real sketches drawn by artists. This gap hinders the generalization of colorization models, meaning they perform poorly on real-world sketches. The paper specifically addresses the risk of this domain gap arising from the resizing step in sketch extraction.
Methods
The core contribution of this paper is the late-resizing strategy. Traditionally, sketch extraction pipelines resize the input color image before detecting lines (early-resizing). The authors argue this early resizing degrades the quality of the color image, leading to broken lines and misleading information for the line detection algorithm.
The proposed late-resizing strategy reverses this order:
- Line Detection: The line detection algorithm operates on the original, high-resolution color image.
- Resizing: The resulting sketch is then resized to the target resolution for the colorization model.
This approach offers several advantages:
- Preserves Color Image Quality: The original color image is not degraded by early downsizing, ensuring better input for line detection.
- Augments Downsizing Artifacts: The resizing of the sketch itself naturally introduces effects like anti-aliasing, mimicking the process of handling real sketches.
- Alleviates Look-Ahead Bias: By maintaining the integrity of the original image information, it reduces the model's tendency to infer incorrect priors based on artifacts.
The authors also introduce a saturation-sensitivity (SS) score as a novel metric to quantitatively evaluate the generalization of colorization models, specifically measuring how well the model activates color for different saturation levels in the sketch.
Datasets
The paper uses the following datasets:
- Safebooru dataset [2]: A large-scale, crowdsourced, and tagged anime illustration dataset used for training and evaluation.
- Webtoon dataset: A custom dataset of approximately 2000 real line-art images collected from Webtoon artists, featuring different painting styles.
Key Insights and Ideas
- The "early-resizing" problem: Resizing an image before line detection is a significant source of domain gap, leading to degraded sketch quality and poor generalization.
- Late-resizing is surprisingly simple yet effective: Reversing the order of resizing and line detection is a straightforward change that yields substantial improvements.
- Quality of the original color image matters: Preserving the fidelity of the input color image is crucial for accurate line detection and, consequently, for better colorization.
- Downsizing as augmentation: The act of resizing the sketch can act as a beneficial augmentation step, mimicking real-world sketch processing.
- New metric for generalization: The saturation-sensitivity score provides a valuable tool to specifically assess a model's ability to generalize to unseen sketches, going beyond traditional metrics like FID.
Key Results
The paper demonstrates the effectiveness of late-resizing through both qualitative and quantitative evaluations.
- Qualitative Improvements: Figures and examples show that late-resizing produces more realistic, continuous lines with adequate thickness and fewer artifacts compared to early-resizing. Colorization results are visibly richer and more detailed, especially on the Safebooru dataset.
- Quantitative Improvements:
- When applied to representative colorization methods like Style2Paints-v3 and AlacGAN, late-resizing consistently improved performance.
- Table 2 shows that late-resizing (marked with "-ours") generally leads to lower FID scores (indicating better performance) and higher SS scores (indicating better generalization) compared to their early-resizing counterparts. For instance, for Style2Paints-v3 on the Safebooru line-art set, the SS score increased from 0.006 to 0.357.
- The saturation-sensitivity score effectively highlights the under-generalization issues of early-resizing.
Limitations and Future Work
- Limited benefit when image size difference is small: The authors acknowledge that the expected benefit of late-resizing might be limited when the difference between the source color image size and the target size is insignificant.
- Future Work: The authors plan to explore super-resolution methods to upscale source color images and integrate them with the late-resizing approach. They also aim to develop further methods to alleviate the ambiguity in sketch extraction strategies for future line-art colorization research.
Summary updated