Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior

Han Wang, Xinning Chai, Yiwen Wang, YuHong Zhang, Rong Xie, Li Song

Introduction

Automatic colorization—the process of restoring color to grayscale images—remains a fundamental challenge in computer vision. Because a single grayscale value can correspond to a vast spectrum of potential colors in the physical world, the task is inherently "ill-posed." Traditional approaches often default to conservative, desaturated predictions, resulting in "muddy" or "washed-out" images. Furthermore, many models suffer from semantic confusion, where a lack of contextual understanding leads to nonsensical color assignments, such as a blue orange or a green sky.

While Generative Adversarial Networks (GANs) and Transformers have made significant strides, they often struggle to balance structural fidelity with the high-fidelity, diverse colorization that human observers expect. This paper introduces the Colorization Diffusion Model (CDM), a novel pipeline that leverages the rich generative priors of pre-trained diffusion models to produce vivid and semantically accurate colorizations.

Methodology

The CDM architecture is designed to bridge the gap between creative generation and structural accuracy through three integrated components: a diffusion prior with luminance guidance, multimodal semantic injection, and a specialized luminance-aware decoder.

1. Diffusion Prior with Luminance Guidance

At its core, the model utilizes a pre-trained Stable Diffusion framework. To ensure that the generated colors remain faithful to the original grayscale structure, the authors inject the grayscale image's latent representation directly into the denoising U-Net at every timestep. This "luminance guidance" acts as a structural anchor, preventing the model from "drifting" or altering the shapes and textures of the original scene during the denoising process.

2. Multimodal Semantic Injection

To resolve the issue of semantic confusion, the CDM incorporates a multimodal semantic generator. By utilizing tools such as BLIP and EfficientNet, the system generates text captions, object categories, and segmentation masks for the input image. These semantic cues are fed into the diffusion process via cross-attention mechanisms. By "reading" that an object is a "sun-ripened orange," the model can apply the correct chromaticity even when the grayscale values are ambiguous.

3. Luminance-Aware Decoder

Standard diffusion decoders often introduce pixel-space distortions when transitioning from latent space back to high-resolution images. To mitigate this, the authors developed a custom luminance-aware decoder. This component utilizes skip connections to pull high-resolution grayscale features directly from the encoder. This ensures that the final output preserves the sharp edges and fine textures of the original input, acting as a corrective lens for the diffusion model’s generative output.

Experimental Results

Datasets and Benchmarks

The CDM was trained on the ImageNet dataset at a 512 × 512 resolution and evaluated using the COCO-Stuff validation set. COCO-Stuff provides a rigorous benchmark, featuring 5,000 images of complex, real-world scenes that require sophisticated contextual understanding to colorize correctly.

Quantitative and Qualitative Performance

The CDM demonstrates superior performance across both automated metrics and human evaluation:

  • Realism (FID): The model achieved a state-of-the-art Fréchet Inception Distance (FID) of 9.799. This significantly outperforms existing diffusion-based methods like ControlNet (10.86) and GAN-based methods like BigColor (which scored above 14), indicating a much higher degree of image realism.
  • Visual Quality: Qualitative analysis shows that the CDM produces more saturated and plausible colors than its competitors. While other models frequently produce "blue shadows" or unnatural skin tones, the CDM maintains color continuity and semantic logic across complex objects.
  • User Preference: In a blind study, participants selected the CDM’s results as their preferred choice 22.59% of the time—the highest rate among all tested automatic methods.

Discussion and Limitations

A key insight of this research is the effective use of "guardrails." By providing the diffusion model with both luminance and semantic constraints, the authors allow the model to utilize its internal knowledge of the world without overwriting the factual data of the source image.

However, the authors note a minor trade-off: the luminance-aware decoder, while excellent for structural fidelity (PSNR), can slightly suppress extreme color saturation to avoid violating the original luminance constraints. Additionally, the reliance on multiple pre-trained models (BLIP, EfficientNet, Transfiner) results in a computationally intensive inference pipeline compared to simpler, end-to-end GAN architectures. Future work may focus on optimizing this pipeline for video colorization, where temporal consistency in both semantics and luminance is paramount.

Conclusion

By combining the generative power of diffusion priors with multimodal semantic guidance and a structure-preserving decoder, the Colorization Diffusion Model sets a new standard for automatic colorization. It successfully addresses the long-standing issues of desaturation and semantic error, moving the field closer to truly automated, high-fidelity photographic restoration.

Summary updated

BibTeX
@inproceedings{wang2024multimodal,
  author = {Han Wang and Xinning Chai and Yiwen Wang and YuHong Zhang and Rong Xie and Li Song},
  title = {{Multimodal Semantic-Aware Automatic Colorization with Diffusion Prior}},
  booktitle = {Workshops @ ICME},
  year = {2024},
  url = {https://openreview.net/forum?id=3qfjCIZIYF},
}
Figure