Presented at IJCAI-25, SeAda (Semantic Adaptation) is a novel framework designed to bridge the "semantic gap" in AI-driven image colorization. By enhancing pre-trained Text-to-Image (T2I) diffusion models, SeAda produces results that are more vivid, semantically accurate, and structurally sound than previous generative methods.
The Challenge: The Semantic Gap
Image colorization is inherently "ill-posed," as a single grayscale image can correspond to multiple valid color versions. While modern diffusion models like Stable Diffusion possess vast prior knowledge, they struggle with semantic discrepancy when applied to grayscale inputs.
When an automated captioner processes a grayscale image, it typically describes only structural elements (e.g., "flowers surrounded by large leaves"). In contrast, captions for color images are much richer (e.g., "red daisies surrounded by green leaves"). Because T2I models were trained on color images with detailed descriptions, they often fail to generate vibrant colors when provided with the sparse semantic information derived from grayscale inputs.
This gap often results in desaturated images or "color bleeding," where colors incorrectly cross object boundaries because the model lacks the semantic cues to distinguish specific color-bearing entities.
The SeAda Framework
The authors introduce two key components to address these issues:
- Semantic Adapter: This module transforms "color-poor" embeddings from grayscale captions into refined, "color-aware" embeddings. Using a cross-attention mechanism, it maps grayscale semantics into the color domain, effectively "hallucinating" plausible color descriptions based on the model's learned priors.
- Controlled T2I Diffusion Model: Built upon Stable Diffusion 1.5, this model utilizes a ControlNet-like structure to preserve spatial integrity. A critical addition is the Representation Cross-Attention (RCA) module, which helps maintain fine-grained structural details during the generative process.
Three-Stage Training Strategy
To maximize synergy between these components, SeAda employs a tiered training approach:
- Stage I: Optimization of the semantic adapter to align grayscale and color embeddings.
- Stage II: Fine-tuning the diffusion model to recover colors using the refined embeddings.
- Stage III: Joint training of both components for peak "semantic adaptability."
Experimental Results and Generalization
SeAda was evaluated against state-of-the-art (SOTA) methods like DDColor and L-CAD on benchmarks including ImageNet, COCO-Stuff, and ADE20K.
- Fidelity: On ImageNet-5K, SeAda achieved a FID (Fréchet Inception Distance) of 3.36, outperforming the previous best of 3.69.
- Vividness: The model showed superior performance in colorfulness difference (ΔCF), producing images closer to natural color distributions.
- Zero-Shot Generalization: On the ADE20K dataset, SeAda improved the FID score by nearly 2 points (7.64 to 5.71) without specific fine-tuning on that dataset.
Qualitative tests on historical photos demonstrate that SeAda avoids the "washed-out" appearance common in earlier models, successfully colorizing complex scenes with high consistency.
Conclusion and Future Directions
While SeAda sets a new standard for semantic accuracy in colorization, challenges remain. The reliance on an external captioner (like CoCa) impacts inference speed, and ensuring historical accuracy for antique photos remains an open problem. Future work may explore "caption-less" semantic adaptation to streamline the workflow while maintaining high-quality output.
Summary updated