When a neural network is tasked with colorizing a grayscale image, it faces a fundamental ambiguity: a single shade of gray could represent a multitude of vibrant colors in the real world. A car in a black-and-white photo might have been bright red, deep blue, or forest green. When models are trained using standard regression objectives, they often produce "muddy," desaturated, or brownish-gray results. This phenomenon is known as regression to the mean.
The Mean Value Trap
The root of the desaturation problem lies in the choice of loss function. Most early colorization models treated the task as a regression problem, typically minimizing the Mean Squared Error (MSE) between predicted and ground-truth pixel values.
As Zhang et al. explain, MSE is minimized by the mathematical mean of a probability distribution. In a color space like $L*a*b*$, where $a$ and $b$ represent color channels, the distribution of possible colors for a given object is often multimodal. If a model is uncertain whether a shirt is red or blue, the MSE-optimal prediction is the average of those two colors—which results in a neutral, desaturated gray. Larsson et al. and Guadarrama et al. note that this "conservative" prediction strategy allows the model to minimize the risk of large errors, but at the cost of the vividness and saturation found in natural scenes.
This effect is not limited to 2D images. Yang et al. observe similar desaturation and blurry boundaries when applying regression objectives to 3D colorization, while Wan identifies this "mean value trap" as a primary hurdle in colorizing monochromatic radiance fields (NeRFs).
From Regression to Classification
To escape the mean value trap, researchers have shifted the problem framing from predicting continuous values to selecting from discrete categories. Zhang et al. proposed treating colorization as a multinomial classification task, dividing the color space into discrete bins. By using a classification loss and rebalancing the importance of rare, high-saturation colors during training, the authors forced the model to commit to vibrant hues rather than hedging its bets with gray.
More recently, Weng et al. extended this idea with CT² (Colorization Transformer), which uses discrete "color tokens." By treating color as a sequence of tokens and applying luminance-based filtering, the authors successfully reduced "color bleeding" and dullness without requiring the complex external priors used in earlier methods.
Generative Priors and Sampling
Another powerful strategy involves using generative models to sample from the distribution of possible colors rather than averaging them. Instead of finding the "most likely" single color, these models learn the underlying manifold of realistic colors.
- Generative Adversarial Networks (GANs): Cao et al. utilized conditional GANs to generate diverse and realistic color schemes, passing human Turing tests where regression models failed. Wu et al. and Kim et al. leveraged pretrained generative priors (like BigGAN) to provide the model with an "internal library" of vivid colors, allowing for semantically accurate and high-contrast results.
- Probabilistic Modeling: Royer et al. and Guadarrama et al. employed autoregressive models (PixelCNN) to "imagine" color palettes pixel-by-pixel, ensuring that each choice is spatially coherent with the last.
- Diffusion Models: The latest shift toward Latent Diffusion Models, as demonstrated by Zabari et al., allows for highly controlled colorization. By diffusing colors in a compressed latent space, these models can produce vibrant, historically accurate restorations guided by granular text prompts.
Semantic and Instance Awareness
Regression to the mean is often exacerbated when a model cannot distinguish between different objects in a scene. If a model sees a "blob" instead of a "person," it defaults to a safe, neutral tone.
To combat this, Chu et al. introduced instance-aware colorization, which detects individual objects and colorizes them independently before fusing them into a global map. Similarly, Jin et al. developed HistoryNet, which uses fine-grained human parsing to restore specific colors to historical attire, such as military uniforms. By providing the network with stronger semantic priors, the authors reduce the ambiguity that leads to statistical averaging.
Even at the architectural level, Kesen et al. showed that using language-conditional filters to modulate both high-level and low-level visual features helps the model "see" specific objects more clearly, further counteracting the tendency to predict frequent, neutral background colors.
Key Foundational Work
PixColor: Pixel Recursive Colorization (2017) — contributed 1 concept elements to this direction.
Learning Representations for Automatic Colorization (2016) — contributed 1 concept elements to this direction.
Probabilistic Image Colorization (2017) — contributed 1 concept elements to this direction.