Stable Diffusion Reference Only: Image Prompt and Blueprint Jointly Guided Multi-Condition Diffusion Model for Secondary Painting

Hao Ai, Lu ShengarXiv 2023 · OpenReview

Abstract

Stable Diffusion and ControlNet have achieved excellent results in the field of image generation and synthesis. However, due to the granularity and method of its control, the efficiency improvement is limited for professional artistic creations such as comics and animation production whose main work is secondary painting. In the current workflow, fixing characters and image styles often need lengthy text prompts, and even requires further training through TextualInversion, DreamBooth or other methods, which is very complicated and expensive for painters. Therefore, we present a new method in this paper, Stable Diffusion Reference Only, a images-to-image self-supervised model that uses only two types of conditional images for precise control generation to accelerate secondary painting. The first type of conditional image serves as an image prompt, supplying the necessary conceptual and color information for generation. The second type is blueprint image, which controls the visual structure of the generated image. It is natively embedded into the original UNet, eliminating the need for ControlNet. We released all the code for the module and pipeline, and trained a controllable character line art coloring model at https://github.com/aihao2000/stable-diffusion-reference-only, that achieved state-of-the-art results in this field. This verifies the effectiveness of the structure and greatly improves the production efficiency of animations, comics, and fanworks.

Introduction

In the professional production of animation and manga, "secondary painting"—the repetitive task of coloring line art and posing characters—is the most labor-intensive part of the workflow. While generative models like Stable Diffusion and ControlNet are powerful, they are primarily text-driven. This creates significant friction for professional artists: unique character features are nearly impossible to describe with words alone, and existing fine-tuning methods like DreamBooth are too resource-intensive for rapid production. Stable Diffusion Reference Only (SDRO) addresses these challenges by replacing complex text prompts with a dual-image guidance system, achieving state-of-the-art coloring precision without per-character fine-tuning.

Methodology

SDRO introduces a multi-condition diffusion model that natively integrates two types of image guidance into the UNet architecture, moving beyond the limitations of text-aligned features.

The Image Prompt and Blueprint

The model utilizes an Image Prompt to provide conceptual and color information. Features are extracted via a CLIP-based Vision Transformer (ViT) and injected into the UNet through a modified cross-attention mechanism.

A reference image providing color and style information.

Simultaneously, a Blueprint provides the structural framework (e.g., line art). Rather than using a heavy external ControlNet branch, SDRO employs a "ControlNet-Lite" approach where a lightweight convolutional network processes the blueprint and embeds it directly into the UNet's latent space. This native embedding reduces computational overhead while maintaining high structural fidelity.

A structural blueprint (line art) extracted for training.

Self-Supervised Training

To train the model to map color distributions from a reference to a structure, the researchers used a self-supervised pipeline. They identified 1,000,000 pairs of highly similar anime images using CLIP. One image in each pair served as the reference, while the other was converted into a "blueprint" using adaptive thresholding and color inversion to simulate hand-drawn line art.

Key Results and Insights

SDRO significantly outperforms existing methods like IP-Adapter and ControlNet Reference Only in maintaining character-specific markers, such as eye color and hair highlights. By training in a purely image-to-image pipeline, the model avoids the "rough" features often produced by text-optimized distributions.

A key finding is the model's ability for Zero-Shot Style Transfer. It can generalize beyond its training data, applying the color palette and shading style of one character to the line art of an entirely different character. This "plug-and-play" capability eliminates the 77-word limit constraints of CLIP-based text prompts and the need for expensive GPU-based fine-tuning.

Discussion and Limitations

While SDRO excels at character coloring, it currently prioritizes character features over complex backgrounds. Additionally, the quality of the final output is sensitive to the precision of the input blueprint; while it supports sketches, clean line art yields the best results. Future exploration could expand the model's training to include photorealistic styles or Western comic aesthetics.

Conclusion

Stable Diffusion Reference Only provides a robust solution for the animation and comic industries by streamlining the secondary painting process. By natively combining image prompts and structural blueprints, it offers a high-efficiency, high-precision tool that significantly reduces the technical barriers for professional artists.

Summary updated

Abstract

Stable Diffusion and ControlNet have achieved excellent results in the field of image generation and synthesis. However, due to the granularity and method of its control, the efficiency improvement is limited for professional artistic creations such as comics and animation production whose main work is secondary painting. In the current workflow, fixing characters and image styles often need lengthy text prompts, and even requires further training through TextualInversion, DreamBooth or other methods, which is very complicated and expensive for painters. Therefore, we present a new method in this paper, Stable Diffusion Reference Only, a images-to-image self-supervised model that uses only two types of conditional images for precise control generation to accelerate secondary painting. The first type of conditional image serves as an image prompt, supplying the necessary conceptual and color information for generation. The second type is blueprint image, which controls the visual structure of the generated image. It is natively embedded into the original UNet, eliminating the need for ControlNet. We released all the code for the module and pipeline, and trained a controllable character line art coloring model at https://github.com/aihao2000/stable-diffusion-reference-only, that achieved state-of-the-art results in this field. This verifies the effectiveness of the structure and greatly improves the production efficiency of animations, comics, and fanworks.
BibTeX
@misc{ai2023stable,
  author = {Hao Ai and Lu Sheng},
  title = {{Stable Diffusion Reference Only: Image Prompt and Blueprint Jointly Guided Multi-Condition Diffusion Model for Secondary Painting}},
  year = {2023},
  eprint = {2311.02343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.02343},
}
Figure