When training a neural network to predict images, video, or color, our natural instinct is to treat the problem as a continuous regression task. After all, light intensity and color wavelengths exist on a continuous spectrum. However, when models are trained using standard regression losses like Mean Squared Error (MSE) or L2 loss, they consistently produce blurry, desaturated, and "muddy" results.
To solve this, a powerful paradigm shift has emerged across computer vision: discretizing continuous visual spaces into quantized bins and treating generation as a multinomial classification task. By forcing networks to choose from a distinct palette of categories rather than predicting a continuous number, models can generate remarkably sharp, vibrant, and realistic visual outputs.
The Trap of Continuous Regression
To understand why discretization works, we must first look at why continuous regression fails in ill-posed visual tasks.
Consider the task of automatic image colorization. If a model is given a grayscale image of an article of clothing, that clothing could plausibly be bright red or bright blue. A regression model optimized via MSE aims to minimize the average squared distance to the true value. Faced with two equally probable but distant possibilities (red and blue), the mathematical way to minimize MSE is to guess the average of those two colors.
The average of bright red and bright blue is a desaturated, muddy gray. Because the model faces this multimodal uncertainty at every single pixel, it defaults to safe, neutral, low-saturation averages. As shown by Richard Zhang et al. in Colorful Image Colorization and Gustav Larsson et al. in Learning Representations for Automatic Colorization, this "mean-seeking" behavior is the primary culprit behind the dull, lifeless outputs of early colorization networks.
The exact same problem plagues video and image generation. If a video model is uncertain whether an object will move left or right in the next frame, an L2 loss forces it to overlay both possibilities, resulting in a blurry blend of both futures.
Embracing Multimodality via Classification
Quantization fundamentally changes how a network handles uncertainty. Instead of outputting a single continuous value, the network outputs a probability distribution over a fixed set of discrete bins using a multi-way softmax layer.
Continuous Regression (MSE):
Bright Red [-----------------------O-----------------------] Bright Blue
Predicted Mean (Dull Gray)
Discrete Classification (Softmax):
[ Bins ] [1] [2] [3] ... [Red Bin] ... [Blue Bin] ... [256]
[ Prob ] 0.0 0.0 0.0 0.48 0.52 0.0
^ ^
No probability assigned to the muddy middle!
By predicting a distribution, the model can assign high probabilities to both the "bright red" bin and the "bright blue" bin simultaneously, while assigning zero probability to the desaturated gray in between. During inference, sampling from this distribution yields a vivid, fully saturated color, completely avoiding the regression-to-the-mean trap.
This insight was leveraged by Aäron van den Oord et al. in Pixel Recurrent Neural Networks. Instead of regressing pixel intensities, they treated each of the 256 possible values of an RGB channel as a discrete category. This allowed the network to capture highly complex, non-parametric distributions—such as assigning high probabilities to pure black and pure white while ignoring intermediate grays—resulting in exceptionally crisp image generations.
Similarly, Nal Kalchbrenner et al. applied this to Video Pixel Networks, demonstrating that modeling pixel intensities as discrete multinomial distributions prevents the blurring of distinct future states over time.
Quantizing the Color Space
For tasks specifically focused on color, operating on 256 raw values per RGB channel can be computationally expensive and perceptually inefficient. Instead, researchers often project images into the Lab color space, where the L channel represents luminance (grayscale structure) and the a and b channels represent color dimensions.
Because human vision is less sensitive to minute gradations in color than to structure, the continuous ab space can be clustered or gridded into a smaller set of discrete representative bins (often around 300 bins).
- Stylistic and Semantic Control: Nanne Van Noord et al. demonstrated in A Learned Representation of Artist-Specific Colourisation that quantizing the color space allows models to learn distinct probability distributions tied to specific artist styles, mapping grayscale inputs to unique "color DNA."
- Simplifying Complex Layouts: In generative tasks where objects must blend into surroundings, framing color as a discrete classification problem simplifies the optimization landscape. For instance, Jeesoo Kim et al. utilized discrete color bins in Vehicle Image Generation Going Well with the Surroundings to synthesize realistic textures within bounding boxes without requiring expensive pixel-wise regression.
Scaling to Modern Tokens and 3D Fields
As generative modeling has transitioned toward Transformer architectures and 3D representations, the core principle of discrete color classification has scaled right along with it.
In the domain of Transformers, Shuchen Weng et al. introduced CT2: Colorization Transformer via Color Tokens. By treating a predefined palette of high-saturation candidates as discrete "color tokens," the transformer can model long-range semantic dependencies across an image, ensuring that an object's color remains consistent and vivid without bleeding into surrounding regions.
This strategy has also proven vital for 3D asset generation. When synthesizing textures for 3D shapes, continuous regression leads to blurry boundaries between different semantic parts of an object. To combat this, Zhenpei Yang et al. developed a method for Learning Generative Neural Networks for 3D Colorization that clusters the Lab color space into discrete bins, yielding significantly sharper boundaries and more vibrant textures.
More recently, this concept was brought to Neural Radiance Fields by Renjie Wan et al. in Colorizing Monochromatic Radiance Fields. By predicting a discrete probability distribution over quantized color categories for 3D points, their framework avoids the desaturated, grayish tones that plague continuous 3D optimization, producing highly saturated and visually consistent 3D scenes.
Ultimately, whether generating a 2D pixel, a 3D radiance field, or selecting structural components from a discrete bank of exemplars—as explored by Xiaohui Zeng et al. in NP-DRAW—abandoning continuous regression in favor of discrete classification allows generative models to preserve the sharp boundaries and rich diversity of the real world.
Key Foundational Work
Colorful Image Colorization (2016) — contributed 2 concept elements to this direction.
Video Pixel Networks (2017) — contributed 2 concept elements to this direction.
Learning Representations for Automatic Colorization (2016) — contributed 2 concept elements to this direction.