TL;DR
This paper introduces Wavelet Transform-assisted Adaptive Generative Modeling (WACM), a novel approach to image colorization that leverages score-based generative models within the wavelet domain. By decomposing images into multi-scale and multi-channel representations, WACM effectively reduces dimensionality and enhances prior learning, leading to significantly improved colorization quality, robustness, and diversity compared to existing methods.
What Problem Does This Paper Address?
Image colorization, the task of adding color to grayscale images, is inherently ill-posed and ambiguous due to the multi-modal nature of the problem. While deep learning, particularly generative models like GANs and VAEs, has shown promise, current methods face limitations. These include the curse of dimensionality in high-resolution images, which hinders score estimation and sampling in models like Noise Conditional Score Networks (NCSN), and the tendency to produce monotonous results. Existing approaches often struggle with capturing rich image priors and achieving diverse, high-quality colorizations, especially for complex, high-resolution images.
Methods
The core innovation of WACM lies in its integration of the Discrete Wavelet Transform (DWT) with score-based generative models.
- Wavelet Domain Modeling: Instead of operating directly in the high-dimensional pixel space, WACM transforms images into the wavelet domain. This decomposition yields multi-scale and multi-channel representations (e.g., 12 channels for RGB images), which are lower-dimensional and more informative. This facilitates more effective prior learning and score estimation.
- Score-Based Generative Model (NCSN): The paper utilizes a denoising score matching (DSM) objective to train a neural network (specifically, a RefineNet architecture) to approximate the score function (gradient of the log probability density) of the data distribution in the wavelet domain.
- Annealed Langevin Dynamics: For generating colorizations, WACM employs annealed Langevin dynamics, a sampling technique that iteratively refines an initial noisy sample using the learned score function.
- Dual Consistency Terms: To guide the generative process and improve colorization quality, two novel consistency terms are introduced:
- Data-Consistency (DC): This term ensures that the generated wavelet coefficients, when transformed back to the image domain, are consistent with the input grayscale image. It minimizes the error between the observed intermediate result and the original input's sub-bands.
- Structure-Consistency (SC): This term addresses issues like "gridding" artifacts and improper color distribution in high-frequency wavelet coefficients. It enforces that the generated high-frequency coefficients maintain a relationship with the original grayscale input's statistics, improving structural integrity and naturalness.
Datasets
The paper uses the following datasets for training and evaluation:
- LSUN (bedroom and church): A large-scale dataset containing labeled images for various scene categories. The indoor LSUN-bedroom and outdoor LSUN-church datasets were used.
- COCO-stuff: A subset of the COCO dataset with 164k images across 172 categories, including things, stuff, and unlabeled classes, often featuring natural scenes.
- CelebA: Used for demonstrating the effectiveness of modeling in the wavelet domain versus intensity domain for generating face images.
No new benchmarks were introduced; the paper focuses on evaluating its method against existing state-of-the-art techniques on these standard datasets.
Key Insights and Ideas
- Wavelet Domain as a Superior Latent Space: The paper strongly advocates for the wavelet domain as a more effective latent space for score-based generative modeling in colorization. The multi-scale, multi-channel representation inherently captures richer image priors and reduces the effective dimensionality, overcoming limitations of pixel-space modeling.
- Dual Consistencies are Crucial: The introduction of Data-Consistency and Structure-Consistency is key to achieving high-quality colorization. DC anchors the generation to the input grayscale, while SC refines the high-frequency details, preventing artifacts and ensuring naturalness.
- Joint Training of Coefficients is Beneficial: Training all wavelet coefficients (low and high frequency) jointly leads to better performance than separate training, suggesting that maintaining consistency across different scales is important for accurate colorization.
- Addressing the "Curse of Dimensionality" Effectively: By transforming to the wavelet domain, the paper demonstrates a practical way to mitigate the challenges posed by high-dimensional data in generative modeling, leading to better performance on higher resolution images.
Key Results
- Quantitative Improvements: WACM consistently outperforms state-of-the-art methods (Zhang et al., MemoPainter, ChromaGAN, iGM-6C) in terms of PSNR and SSIM on 128x128 images, achieving the highest scores on LSUN-church and LSUN-bedroom datasets.
- Qualitative Superiority: Visual comparisons show WACM producing more natural, vibrant, and artifact-free colorizations, especially for higher resolution images (128x128 and 256x256), where naive NCSN struggles. The "gridding" artifacts are significantly reduced.
- Robustness: A single model trained on COCO-stuff can effectively colorize diverse real-world images, including legacy photos and cartoons, demonstrating good generalization.
- Efficiency: While iterative, WACM's runtime per iteration is competitive, and the wavelet domain approach potentially speeds up convergence.
Limitations and Future Work
- Colorization Degradation due to Data Distribution: WACM's performance can degrade on images whose colored versions fall into low-density regions of the learned distribution, especially if they differ significantly from the training data.
- Computational Burden: The iterative nature of score-based generative models, while powerful, can be computationally intensive compared to some supervised methods.
- Future Work:
- Few-Shot Colorization: Extending the method to few-shot colorization tasks.
- Meta-Learning and Clustering: Incorporating meta-learning and clustering for more robust and data-efficient colorization.
- Multi-Level Wavelet Priors: Further exploring multi-level wavelet transform priors.
- Hyperparameter Optimization: Investigating advanced statistical methods like Bayesian optimization for hyperparameter tuning.
- Parallelization: Exploring parallelization strategies to reduce computation time.
Summary updated