Probabilistic Image Colorization

Amelie Royer, Alexander Kolesnikov, Christoph H. Lampert

Introduction

Colorizing a grayscale image is a classic "ill-posed" problem in computer vision. While some elements like the sky or grass have predictable colors, many objects—such as clothing or vehicles—could plausibly take on a wide variety of hues. Traditional deep learning approaches typically treat this as a regression task, training feed-forward networks to predict a single color value per pixel. This often results in two significant drawbacks: a lack of diversity in output and a "grayish" effect, where the model predicts the mathematical mean of all possible colors, leading to desaturated results.

The Probabilistic Image Colorization (PIC) Framework

To address these issues, researchers from IST Austria proposed the Probabilistic Image Colorization (PIC) model. Instead of seeking a single "best guess," PIC models the joint distribution of colors across the entire image. This allows the system to capture the inherent uncertainty of the task and generate multiple, diverse, and spatially coherent colorizations for a single input.

The architecture is divided into two primary components:

  1. Embedding Network ($g^w$): A deep feed-forward network that processes the grayscale input to extract high-level semantic information. This "map" informs the model about the objects present in the scene (e.g., identifying a bird versus a tree).
  2. Autoregressive Network ($f^\theta$): Based on the PixelCNN++ architecture, this component predicts the color of each pixel sequentially. Crucially, each prediction is conditioned on both the semantic embedding and the colors of previously processed pixels.
The PIC architecture overview, showing the transition from grayscale input to a sampled colorization via an autoregressive distribution prediction.

To manage the high dimensionality of the color space (65,536 combinations in LAB space), the authors utilize a mixture of 10 logistic distributions. This probabilistic approach allows the model to represent multimodality—the existence of several distinct, valid color options—while remaining computationally efficient.

Key Insights and Methodology

A central innovation of PIC is its focus on the joint distribution rather than independent pixel classification. By modeling the relationships between neighboring pixels, the network naturally ensures spatial coherence without requiring complex post-processing steps.

Furthermore, the "grayish" problem is solved through sampling. Because the model learns a full probability distribution, users can sample from it to produce vivid, saturated colors. In one sample, a car might be rendered as vibrant red; in another, a deep blue. This eliminates the need for the manual "class-rebalancing" hacks used in previous state-of-the-art models to force vibrancy.

Examples of sampled chrominance maps ($X^{ab}$) for a rabbit and a person, illustrating the model's ability to generate vivid and diverse color distributions.

Experimental Results

The model was evaluated on the CIFAR-10 and ILSVRC 2012 (ImageNet) datasets. On ImageNet, images were rescaled to 128x128 to balance computational demands with visual detail.

  • Quantitative Performance: Using Negative Log-Likelihood (NLL) as a metric for distribution accuracy, PIC achieved 2.72 bits-per-dimension on CIFAR-10 and 2.51 on ImageNet.
  • Vibrancy and Saturation: In ablation studies, the full PIC model reached an average perceptual saturation of 36.4%, significantly outperforming versions of the model that lacked the autoregressive component (32.7%).
  • Qualitative Diversity: The model successfully generates multiple plausible color schemes for complex subjects, such as varying the plumage of a bird while maintaining realistic textures.

Limitations and Future Directions

While PIC represents a significant step forward, challenges remain. The autoregressive nature of the model makes it computationally expensive to scale to ultra-high resolutions. Additionally, the model can occasionally struggle with long-range dependencies, such as maintaining color consistency across an object that is partially occluded. Future work may focus on improving semantic understanding in complex scenes and optimizing the architecture for larger image sizes.

Conclusion

Probabilistic Image Colorization moves the field beyond deterministic "best guesses" toward a more nuanced understanding of visual uncertainty. By modeling the joint color distribution, PIC provides a principled way to generate diverse, vivid, and natural-looking images from grayscale inputs.

Summary updated

BibTeX
@inproceedings{royer2017probabilistic,
  author = {Amelie Royer and Alexander Kolesnikov and Christoph H. Lampert},
  title = {{Probabilistic Image Colorization}},
  booktitle = {BMVC},
  year = {2017},
  url = {https://openreview.net/forum?id=SutYRD4Gobyj},
}
Figure