Abstract
Language-based colorization aims to transform grayscale images into color versions guided by natural language descriptions. Traditional methods often struggle with instance confusion—where multiple similar objects are incorrectly colorized—and luminance bias, where models rely on brightness rather than text. We present L-CoIns (Language-based Colorization with Instance awareness), a novel transformer-based framework that achieves precise, text-driven colorization. By utilizing a grouping mechanism for unsupervised instance detection and luminance augmentation to decouple color from brightness, L-CoIns sets a new state-of-the-art in instance-aware colorization without requiring external object detectors.
Introduction
The task of language-based colorization is inherently challenging due to the ambiguity of mapping a single luminance channel to three color channels. Users expect the ability to specify colors for individual objects, such as "the woman on the left wears a blue shirt." However, existing models face two significant hurdles:
- Instance Confusion: When a prompt describes different colors for multiple instances of the same class (e.g., two people), models often fail to distinguish between them, leading to uniform colorization or color bleeding. Most current solutions rely on pre-trained object detectors, which are limited by their training data and can fail on out-of-distribution images.
- Luminance Bias: Models frequently learn statistical shortcuts, associating specific grayscale intensities with certain colors (e.g., dark pixels as blue, bright as yellow). This "lazy" learning causes models to ignore textual instructions that contradict these correlations.
Methodology
L-CoIns introduces several structural innovations to move beyond simple feature fusion and toward a deeper understanding of image composition.
Grouping Transformer
To achieve instance awareness without external priors, L-CoIns employs a Grouping Transformer. This component uses learnable "group tokens" that automatically aggregate similar image patches. This unsupervised mechanism allows the model to naturally segment distinct instances, enabling it to associate specific words in a prompt with the correct regions of the image.
Luminance Augmentation
To address luminance bias, the researchers implemented Luminance Augmentation. By randomly rotating hues in the HSV space and applying gamma corrections during training, the model is forced to rely on the text prompt rather than the grayscale value to determine the correct color. This effectively decouples brightness from color information.
Counter-color Loss
A novel Counter-color Loss is used during training. The model is presented with prompts where color terms are intentionally swapped. It is then penalized if it cannot distinguish between the ground truth and the "counter-color" version, reinforcing the model's adherence to descriptive consistency.
Experimental Results
The performance of L-CoIns was evaluated on the Extended COCO-Stuff benchmark and a newly curated Multi-instance Dataset containing 72,000 images with detailed, instance-specific descriptions.
Quantitative Performance
L-CoIns significantly outperformed previous state-of-the-art models, including L-CoDe and L-CoDer. On the Multi-instance dataset, L-CoIns achieved a PSNR of 24.823 and an SSIM of 0.917, indicating superior reconstruction quality and structural similarity.
Qualitative Analysis and User Study
Qualitatively, L-CoIns demonstrated a unique ability to handle "statistically unlikely" requests, such as colorizing a dark object with a bright color (e.g., a dark yellow cup). It also showed remarkable robustness when applied to historical black-and-white photography. In a human preference study, L-CoIns was chosen as the most natural-looking result in 25.4% of cases, outperforming the next best model (21.3%).
Conclusion and Future Work
L-CoIns represents a significant step forward in controllable image colorization. By integrating unsupervised instance grouping and breaking luminance-color correlations, it provides a more flexible and accurate tool for text-driven editing. While the model currently struggles with very small objects in dense, complex captions, future iterations may incorporate unsupervised "zoom-in" mechanisms to further refine detail without sacrificing the benefits of an end-to-end transformer architecture.
Summary updated