When we look at a grayscale photograph of an apple, our brains instantly fill in the blanks: it could be a vibrant red, a crisp green, or a sun-kissed yellow. However, to a computer vision model, this choice represents a fundamental mathematical crisis known as an ill-posed inverse problem.
Because a single luminance (brightness) value can map to an infinite variety of valid chrominance (color) values, mapping grayscale to color is a one-to-many problem. If you train a neural network using standard regression losses—like Mean Squared Error (L2)—the network attempts to minimize its risk by averaging all possible valid colors together. The mathematical average of red, green, and yellow is a muddy, desaturated gray-brown. Consequently, early automatic colorization models consistently produced washed-out, conservative results.
To solve this chromatic ambiguity, researchers had to stop treating colorization as a simple regression task and instead design architectures capable of modeling complex, multimodal probability distributions.
From Regression to Multinomial Classification
The first major breakthrough in overcoming the "regression to the mean" trap was to reframe colorization as a classification problem. Instead of asking a network to predict the exact continuous $a$ and $b$ channels of the Lab color space, why not ask it to choose from a grid of discrete color buckets?
Grayscale Input ──> [ Deep Semantic Features ] ──> Probability Distribution over Color Bins ──> Class Rebalancing ──> Vibrant Output
In Colorful Image Colorization, Richard Zhang et al. divided the color space into discrete bins and trained a network to output a probability distribution over these bins for every pixel. Crucially, the authors introduced class rebalancing during training. Because natural images are dominated by low-chromance background colors like skies, roads, and walls, rare and vibrant colors are heavily penalized by standard cross-entropy losses. By reweighting the loss function to favor rare color bins, the model was encouraged to make bold, saturated color choices.
Simultaneously, Gustav Larsson et al. demonstrated in Learning Representations for Automatic Colorization that predicting per-pixel color histograms allowed the network to retain ambiguity until the final prediction layer. Instead of forcing a pixel to commit to a single color early on, the model preserves the entire distribution of possibilities, leveraging deep semantic features to resolve what color an object should be based on its identity.
Enforcing Spatial Coherence Autoregressively
While per-pixel classification prevents muddy colors, it introduces a new problem: spatial inconsistency. If a model predicts the color of each pixel independently, neighboring pixels might sample from different modes of the distribution, leading to a pixelated, disjointed mess where a car is half-red and half-blue.
To ensure that color choices are globally and locally coherent, researchers turned to autoregressive modeling. The intuition here is simple: the color of the current pixel should depend heavily on the colors of the pixels that came before it.
In PixColor: Pixel Recursive Colorization, Sergio Guadarrama et al. split the task into two distinct stages. First, a probabilistic PixelCNN "imagines" a low-resolution color palette by conditioning each pixel's color on its spatial predecessors. Because the model generates colors recursively, it ensures that if the front of an apple is colored red, the rest of the apple follows suit. A second refinement network then upsamples this low-resolution palette into a sharp, high-resolution final image.
Similarly, Amelie Royer et al. utilized this autoregressive framework in Probabilistic Image Colorization to generate multiple, highly diverse colorizations for a single grayscale image. By sampling different trajectories from the learned conditional probability distribution, the model can generate one version of a scene where a jacket is blue, and another where it is yellow, with both images remaining perfectly coherent internally.
Anchoring Color in Generative Priors
As generative modeling advanced, researchers realized they didn't need to teach colorization networks the rules of the visual world from scratch. Instead, they could exploit the rich, pre-existing "internal libraries" of colors stored inside large-scale pretrained models.
Yanze Wu et al. pioneered this approach in Towards Vivid and Diverse Image Colorization with Generative Color Prior by leveraging a pretrained Generative Adversarial Network (GAN). Instead of directly generating pixels, the colorization network learns to navigate the latent space of the GAN, finding the exact latent code that matches the semantic structure of the grayscale input. Because the GAN's latent space is inherently constrained to produce realistic, vibrant images, the model completely bypasses the risk of producing desaturated or muddy artifacts.
However, standard GAN latents often struggle with complex spatial layouts. To fix this, Geonung Kim introduced BigColor: Colorization Using a Generative Color Prior for Natural Images, which replaces flat latent codes with high-dimensional spatial feature maps. This allows the generative prior to colorize intricate structures and complex, multi-object scenes without losing spatial fidelity.
Today, this paradigm has shifted toward diffusion models. In Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior, Han Wang et al. demonstrated that conditioning a diffusion process on a luminance-aware decoder allows the model to iteratively refine colors. The diffusion prior acts as a powerful regularizer: whenever the colorization process begins to drift toward an unrealistic or desaturated state, the diffusion model pulls the trajectory back toward a semantically plausible, vivid mode.
Collapsing Ambiguity with Multimodal Guidance
Even with powerful generative priors, some ambiguities cannot be resolved automatically. For instance, no algorithm can look at a black-and-white photo of a historical figure and know whether their tie was blue or burgundy. In these scenarios, the only way to resolve the one-to-many mapping is to introduce external constraints or user intent.
One intuitive way to guide a model is through natural language. In Coloring with Words: Guiding Image Colorization Through Text-Based Palette Generation, Hyojin Bahng et al. introduced a two-stage framework that translates text prompts (like "rainforest" or "furious heart") into a discrete 5-color palette. The model then uses this palette as a spatial constraint, mapping the chosen colors directly onto the corresponding semantic regions of the grayscale image.
Text Prompt ──> [ Palette Generator ] ──> 5-Color Palette ──> [ Colorization Network ] ──> Guided Output
To handle these complex, conditional relationships across an entire image, modern architectures rely heavily on self-attention mechanisms. Manoj Kumar and Dirk Weissenborn developed the Colorization Transformer, which uses axial self-attention to dynamically relate distant parts of an image. If a user provides a single color hint or text prompt, the transformer propagates that color choice across all semantically related pixels, ensuring that the entire scene adapts fluidly to resolve the user's stylistic intent.
By shifting from rigid pixel-level regression to probabilistic classification, autoregressive consistency, and generative priors, modern colorization systems no longer guess a single "average" color. Instead, they understand the full spectrum of what could be, choosing a single, vivid reality from an ocean of possibilities.
Key Foundational Work
Local linear model optimization based grayscale image colorization (2013) — contributed 1 concept elements to this direction.