Line Art Correlation Matching Feature Transfer Network for Automatic Animation Colorization

Qian Zhang, Bo Wang, Wei Wen, Hai Li, Junhui LiuWACV 2021

TL;DR

This paper introduces a novel network, CMFTN, for automatically colorizing animation line art sequences. It addresses the critical challenge of maintaining color consistency between frames, even with significant motion, by proposing a "correlation matching feature transfer" mechanism to align features from previous frames to the current one. This approach significantly improves the quality and temporal coherence of colorized animation frames.

What Problem Does This Paper Address?

Coloring animation frames is a labor-intensive and time-consuming process. While existing image-to-image translation methods using Generative Adversarial Networks (GANs) can colorize single line art images effectively, they struggle with animation sequences. The core problem is maintaining in-between frame consistency. Simply using the previous colored frame as a reference for the next line art frame leads to errors because the spatial alignment between the previous colored frame and the current line art sketch is often imperfect, especially when characters or scenes change. This misalignment can "mislead" the colorization process. Furthermore, these methods often ignore the information from the previous sketch frame, which is crucial for understanding temporal continuity.

Methods

The paper proposes a Line Art Correlation Matching Feature Transfer Network (LCMFTN). The key innovation is the Correlation Matching Feature Transfer (CMFT) model. This model aims to align the colored reference feature (from the previous frame) to the current frame's features in a learnable way. It leverages two types of consistency:

  1. Domain Style Consistency: The inherent similarity in style between consecutive line art frames.
  2. Spatial Content Consistency: The assumption that motion in the line art is maintained across color domains.

The CMFT model works by matching correlations in feature maps. It takes features from the current sketch, the previous sketch, and the previous colored frame as input. It then generates a mapping matrix to warp the previous colored frame's features to align with the current sketch's spatial domain. This aligned feature is then used by a U-Net based generator to reconstruct the current colored frame.

The LCMFTN integrates multiple CMFT models within a coarse-to-fine decoder architecture. This allows for progressive refinement of the colorization, starting with global color composition and moving towards finer details. To handle the sparsity of line art, they also introduce a network to reduce the matching difficulty. The generator uses ResNeXt blocks for increased capacity and sub-pixel convolution layers for upsampling.

Datasets

The authors collected a dataset of 10 cartoon films from Hayao Miyazaki. Three films were used for training, and the remaining seven for testing. To create training data, they extracted frame pairs from the same "shot" to ensure temporal coherence. They simulated line art sketches by applying a tool called "paintchainer's LeNet" to the colored frames.

Key Insights and Ideas

  • Temporal consistency is paramount for animation: Unlike single image tasks, animation requires a strong understanding of how frames evolve over time.
  • Spatial misalignment is a major hurdle: Directly using previous colored frames as references is problematic due to imperfect alignment, especially with motion.
  • Correlation matching is key to alignment: The CMFT model's ability to learn feature correlations allows for robust alignment of temporal information, even with significant changes.
  • Coarse-to-fine generation improves results: Building up the colorization from global to local details, guided by aligned features at each stage, leads to better overall quality.
  • Leveraging both previous sketch and color frames is beneficial: The model explicitly uses both the previous sketch and its corresponding color frame to inform the current frame's colorization.

Key Results

  • The proposed LCMFTN significantly improves in-between frame consistency and the quality of colored frames, especially when motion is intense and diverse.
  • Quantitative results (PSNR/SSIM) presented in Tables 1-3 show that LCMFTN consistently outperforms other methods like TCVC, Pix2Pix, and DeepAnalogy, particularly as the frame interval (stride) increases, indicating better handling of larger temporal gaps and motion.
  • Qualitative results (Figures 5-7) visually demonstrate the superior temporal coherence and color fidelity of LCMFTN compared to baselines. For instance, Figure 5 shows that LCMFTN produces more consistent colors across frames than TCVC, which exhibits color instability.

Limitations and Future Work

  • Computational Cost: The paper notes that the CMFT model's correlation matrix can be very large (65536x65536), leading to significant memory overhead and slow training/inference. They address this by removing the CMFT³ model in their implementation.
  • Dataset Bias: While they collected a diverse dataset, the focus on specific anime films might limit generalizability to other animation styles.
  • Future Work: The paper doesn't explicitly detail future work, but the inherent challenges of handling extreme motion, complex character interactions, and diverse artistic styles in animation colorization offer ample avenues for further research. Improving the efficiency of the CMFT model and exploring more sophisticated feature alignment techniques would be valuable.

Summary updated

BibTeX
@inproceedings{zhang2021line,
  author = {Qian Zhang and Bo Wang and Wei Wen and Hai Li and Junhui Liu},
  title = {{Line Art Correlation Matching Feature Transfer Network for Automatic Animation Colorization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure