Pro-CCaps: Progressively Teaching Colourisation to Capsules

Rita Pucci, Christian Micheloni, Gian Luca Foresti, Niki MartinelWACV 2022

TL;DR

This paper introduces Pro-CCaps, a novel deep learning architecture for automatic image colorization that effectively combines global image features with fine-grained entity-level features. By employing a progressive learning strategy, Pro-CCaps gradually increases its complexity during training, leading to more stable and vibrant colorizations compared to existing methods.

What Problem Does This Paper Address?

Automatic image colorization aims to restore color to grayscale images. A key challenge in this task is the inherent multimodality of color: the same object can appear in different colors, and the same color can be associated with different objects or properties. Existing approaches often struggle with this by either focusing solely on global image features (leading to unsaturated colors for specific objects) or by treating entity-level features separately, failing to integrate them effectively. This can result in colorizations that lack vibrancy, have smudges on object boundaries, or misattribute colors. The paper addresses this by proposing a method that fosters a strong collaboration between image-level and entity-level features.

Methods

The core of Pro-CCaps lies in its architecture and training strategy.

  • Architecture: It combines convolutional layers for extracting image-level features with capsule layers for identifying entity-level features. These two types of features are fused through residual and skip connections, promoting a collaborative learning process. The architecture consists of a downsampling phase (feature extraction) and an upsampling phase (feature reconstruction).
  • Progressive Growing While Learning (ProGL): To overcome the issue of global features dominating entity-level ones and to manage the complexity of high-resolution image reconstruction, the authors adapt the ProGL strategy. Instead of training the entire network end-to-end, Pro-CCaps starts with a simpler network and progressively adds layers and complexity during training. This allows the model to learn coarse structures first and then refine finer details, building upon prior knowledge from earlier stages. This is applied to the upsampling phase.
  • Channels Reconstruction: A dedicated block reconstructs the chrominance (a,b) channels of the CIELab color space. It uses both a quantized representation (to ensure colors are within a valid gamut and avoid averaging effects) and a chroma representation (to predict consistent colors).

Datasets

The paper uses the following datasets:

  • ImageNet10k: Used for testing and evaluation.
  • COCOStuff: Used for training and fine-tuning.
  • Places205: Used for evaluation.
  • DIV2K: Used for comparison with specific prior works.

Key Insights and Ideas

  • Entity-level features are crucial: The paper strongly advocates for the importance of identifying and integrating entity-level features, not just global image context, for realistic colorization.
  • Collaboration over separation: Simply extracting image-level and entity-level features separately and then merging them is suboptimal. A true collaboration and interaction between these feature types are necessary.
  • Progressive learning stabilizes training: Applying ProGL to the upsampling phase helps manage the complexity of reconstructing high-resolution images and leads to more stable training and better feature integration. This is particularly important for preventing early layers from being overwhelmed by the reconstruction task.
  • Metrics vs. Human Perception: The study highlights a potential discrepancy between standard quantitative metrics (like PSNR, SSIM, LPIPS) and human perception of colorization quality, with users often preferring the more vibrant results from Pro-CCaps even when metrics might favor other models.

Key Results

  • Quantitative Performance: Pro-CCaps outperforms state-of-the-art methods on the ImageNet10k dataset across PSNR, SSIM, and LPIPS metrics. It achieves competitive results on other datasets as well.
  • Qualitative Improvements: Visual results show that Pro-CCaps produces more saturated, vivid, and consistent colorizations, with sharper object boundaries and fewer smudges compared to baseline methods.
  • User Study: A large-scale user study revealed that users preferred Pro-CCaps' colorizations 73% of the time over a strong baseline (CCaps), indicating a better perceptual realism.
  • Parameter Efficiency: Pro-CCaps uses significantly fewer trainable parameters than some comparable methods (e.g., about one-third of the parameters used by Su et al. [22]).

Limitations and Future Work

  • Metric Discrepancy: The paper acknowledges a discrepancy between quantitative metrics and user preference, suggesting that current metrics might not fully capture the perceptual quality of colorization. The authors suggest that community efforts are needed to develop better evaluation metrics.
  • Future Work: While not explicitly detailed, the success of ProGL in stabilizing and improving colorization suggests its potential application in other generative tasks that involve progressive learning and feature fusion. The architecture's ability to integrate different feature types also opens avenues for exploring more complex scene understanding tasks.

Summary updated

BibTeX
@inproceedings{pucci2022pro,
  author = {Rita Pucci and Christian Micheloni and Gian Luca Foresti and Niki Martinel},
  title = {{Pro-CCaps: Progressively Teaching Colourisation to Capsules}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure