L-CoIns: Language-based Colorization With Instance Awareness

Zheng Chang, Shuchen Weng, Peixuan Zhang, Yu Li, Si Li, Boxin Shi

Abstract

Language-based colorization aims to transform grayscale images into color versions guided by natural language descriptions. Traditional methods often struggle with instance confusion—where multiple similar objects are incorrectly colorized—and luminance bias, where models rely on brightness rather than text. We present L-CoIns (Language-based Colorization with Instance awareness), a novel transformer-based framework that achieves precise, text-driven colorization. By utilizing a grouping mechanism for unsupervised instance detection and luminance augmentation to decouple color from brightness, L-CoIns sets a new state-of-the-art in instance-aware colorization without requiring external object detectors.

Introduction

The task of language-based colorization is inherently challenging due to the ambiguity of mapping a single luminance channel to three color channels. Users expect the ability to specify colors for individual objects, such as "the woman on the left wears a blue shirt." However, existing models face two significant hurdles:

  1. Instance Confusion: When a prompt describes different colors for multiple instances of the same class (e.g., two people), models often fail to distinguish between them, leading to uniform colorization or color bleeding. Most current solutions rely on pre-trained object detectors, which are limited by their training data and can fail on out-of-distribution images.
  2. Luminance Bias: Models frequently learn statistical shortcuts, associating specific grayscale intensities with certain colors (e.g., dark pixels as blue, bright as yellow). This "lazy" learning causes models to ignore textual instructions that contradict these correlations.
Grayscale input image featuring multiple instances of the same object class (women).

Methodology

L-CoIns introduces several structural innovations to move beyond simple feature fusion and toward a deeper understanding of image composition.

Grouping Transformer

To achieve instance awareness without external priors, L-CoIns employs a Grouping Transformer. This component uses learnable "group tokens" that automatically aggregate similar image patches. This unsupervised mechanism allows the model to naturally segment distinct instances, enabling it to associate specific words in a prompt with the correct regions of the image.

Luminance Augmentation

To address luminance bias, the researchers implemented Luminance Augmentation. By randomly rotating hues in the HSV space and applying gamma corrections during training, the model is forced to rely on the text prompt rather than the grayscale value to determine the correct color. This effectively decouples brightness from color information.

Counter-color Loss

A novel Counter-color Loss is used during training. The model is presented with prompts where color terms are intentionally swapped. It is then penalized if it cannot distinguish between the ground truth and the "counter-color" version, reinforcing the model's adherence to descriptive consistency.

Colorization result by L-CoIns, demonstrating instance-aware color application based on textual prompts.

Experimental Results

The performance of L-CoIns was evaluated on the Extended COCO-Stuff benchmark and a newly curated Multi-instance Dataset containing 72,000 images with detailed, instance-specific descriptions.

Quantitative Performance

L-CoIns significantly outperformed previous state-of-the-art models, including L-CoDe and L-CoDer. On the Multi-instance dataset, L-CoIns achieved a PSNR of 24.823 and an SSIM of 0.917, indicating superior reconstruction quality and structural similarity.

Qualitative Analysis and User Study

Qualitatively, L-CoIns demonstrated a unique ability to handle "statistically unlikely" requests, such as colorizing a dark object with a bright color (e.g., a dark yellow cup). It also showed remarkable robustness when applied to historical black-and-white photography. In a human preference study, L-CoIns was chosen as the most natural-looking result in 25.4% of cases, outperforming the next best model (21.3%).

Conclusion and Future Work

L-CoIns represents a significant step forward in controllable image colorization. By integrating unsupervised instance grouping and breaking luminance-color correlations, it provides a more flexible and accurate tool for text-driven editing. While the model currently struggles with very small objects in dense, complex captions, future iterations may incorporate unsupervised "zoom-in" mechanisms to further refine detail without sacrificing the benefits of an end-to-end transformer architecture.

Summary updated

Abstract

Language-based colorization produces plausible colors consistent with the language description provided by the user. Recent studies introduce additional annotation to prevent color-object coupling and mismatch issues, but they still have difficulty in distinguishing instances corresponding to the same object words. In this paper, we propose a transformer-based framework to automatically aggregate similar image patches and achieve instance awareness without any additional knowledge. By applying our presented luminance augmentation and counter-color loss to break down the statistical correlation between luminance and color words, our model is driven to synthesize colors with better descriptive consistency. We further collect a dataset to provide distinctive visual characteristics and detailed language descriptions for multiple instances in the same image. Extensive experiments demonstrate our advantages of synthesizing visually pleasing and description-consistent results of instance-aware colorization.
BibTeX
@inproceedings{chang2023l,
  author = {Zheng Chang and Shuchen Weng and Peixuan Zhang and Yu Li and Si Li and Boxin Shi},
  title = {{L-CoIns: Language-based Colorization With Instance Awareness}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01842},
}
Figure