Unleashing the Semantic Adaptability of Controlled Diffusion Model for Image Colorization

Xiangcheng Du, Zhao Zhou, Yanlong Wang, Yingbin Zheng, Xingjiao Wu, Peizhu Gong, Cheng JinIJCAI 2025 · DOI

Presented at IJCAI-25, SeAda (Semantic Adaptation) is a novel framework designed to bridge the "semantic gap" in AI-driven image colorization. By enhancing pre-trained Text-to-Image (T2I) diffusion models, SeAda produces results that are more vivid, semantically accurate, and structurally sound than previous generative methods.

The Challenge: The Semantic Gap

Image colorization is inherently "ill-posed," as a single grayscale image can correspond to multiple valid color versions. While modern diffusion models like Stable Diffusion possess vast prior knowledge, they struggle with semantic discrepancy when applied to grayscale inputs.

When an automated captioner processes a grayscale image, it typically describes only structural elements (e.g., "flowers surrounded by large leaves"). In contrast, captions for color images are much richer (e.g., "red daisies surrounded by green leaves"). Because T2I models were trained on color images with detailed descriptions, they often fail to generate vibrant colors when provided with the sparse semantic information derived from grayscale inputs.

The semantic gap: grayscale images generate sparse captions compared to their color counterparts, leading to desaturated outputs in standard models.

This gap often results in desaturated images or "color bleeding," where colors incorrectly cross object boundaries because the model lacks the semantic cues to distinguish specific color-bearing entities.

The SeAda Framework

The authors introduce two key components to address these issues:

  1. Semantic Adapter: This module transforms "color-poor" embeddings from grayscale captions into refined, "color-aware" embeddings. Using a cross-attention mechanism, it maps grayscale semantics into the color domain, effectively "hallucinating" plausible color descriptions based on the model's learned priors.
  2. Controlled T2I Diffusion Model: Built upon Stable Diffusion 1.5, this model utilizes a ControlNet-like structure to preserve spatial integrity. A critical addition is the Representation Cross-Attention (RCA) module, which helps maintain fine-grained structural details during the generative process.

Three-Stage Training Strategy

To maximize synergy between these components, SeAda employs a tiered training approach:

  • Stage I: Optimization of the semantic adapter to align grayscale and color embeddings.
  • Stage II: Fine-tuning the diffusion model to recover colors using the refined embeddings.
  • Stage III: Joint training of both components for peak "semantic adaptability."

Experimental Results and Generalization

SeAda was evaluated against state-of-the-art (SOTA) methods like DDColor and L-CAD on benchmarks including ImageNet, COCO-Stuff, and ADE20K.

  • Fidelity: On ImageNet-5K, SeAda achieved a FID (Fréchet Inception Distance) of 3.36, outperforming the previous best of 3.69.
  • Vividness: The model showed superior performance in colorfulness difference (ΔCF), producing images closer to natural color distributions.
  • Zero-Shot Generalization: On the ADE20K dataset, SeAda improved the FID score by nearly 2 points (7.64 to 5.71) without specific fine-tuning on that dataset.

Qualitative tests on historical photos demonstrate that SeAda avoids the "washed-out" appearance common in earlier models, successfully colorizing complex scenes with high consistency.

Conclusion and Future Directions

While SeAda sets a new standard for semantic accuracy in colorization, challenges remain. The reliance on an external captioner (like CoCa) impacts inference speed, and ensuring historical accuracy for antique photos remains an open problem. Future work may explore "caption-less" semantic adaptation to streamline the workflow while maintaining high-quality output.

Summary updated

BibTeX
@inproceedings{du2025unleashing,
  author = {Xiangcheng Du and Zhao Zhou and Yanlong Wang and Yingbin Zheng and Xingjiao Wu and Peizhu Gong and Cheng Jin},
  title = {{Unleashing the Semantic Adaptability of Controlled Diffusion Model for Image Colorization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/106},
}
Figure