JieHua Paintings Style Feature Extracting Model using Stable Diffusion with ControlNet

Yujia Gu, Haofeng Li, Xinyu Fang, Zihan Peng, Yinan PengarXiv 2024 · DOI · arXiv

Abstract

Style transfer in traditional Chinese landscape painting, particularly the architectural genre known as Jiehua, presents unique challenges in balancing global artistic texture with local structural precision. This paper introduces a Fine-tuned Stable Diffusion Model with ControlNet (FSDMC) designed to extract and apply these intricate stylistic features. By utilizing Canny edge detection for spatial conditioning and specific text prompts, the proposed model achieves a Fréchet Inception Distance (FID) of 3.27, vastly outperforming the industry-standard CycleGAN. Qualitative evaluations by experts further confirm the model's ability to preserve semantic information while accurately rendering the geometric hallmarks of Jiehua.

Introduction

Jiehua, or "boundary painting," is a sub-genre of Chinese painting characterized by the precise, ruler-assisted depiction of buildings and structures. Unlike more fluid ink-wash styles, Jiehua requires high geometric accuracy. Traditional style transfer methods, such as CycleGAN, often suffer from "catastrophic forgetting," where the semantic content of the original image is lost, or structural details are warped. There is a need for a generative approach that can capture the specific "fingerprints" of Jiehua artists—such as Yao Wen Han—without compromising the integrity of the scene's architecture.

Methodology

The researchers compared two primary architectures for this task: a baseline CycleGAN and the proposed FSDMC.

CycleGAN Baseline

CycleGAN is an unpaired image-to-image translation model utilizing dual generators and discriminators. It relies on cycle consistency loss to ensure that an image translated from Domain A to B can be mapped back to A. While effective for general texture transfer, it often fails to maintain the rigid structural lines essential to Jiehua.

FSDMC: Stable Diffusion with ControlNet

The proposed FSDMC leverages Stable Diffusion v1.5 augmented with ControlNet to provide fine-grained spatial control.

Architecture of the ControlNet integration with Stable Diffusion, showing the locked and trainable blocks connected via zero convolutions.

As shown in the architecture diagram above, the model employs several key mechanisms:

  • Locked vs. Trainable Blocks: The pre-trained Stable Diffusion weights are "locked" to retain broad generative knowledge, while a "trainable copy" is fine-tuned on the Jiehua dataset.
  • Zero Convolutions: These layers connect the ControlNet to the main model. Initialized at zero, they allow the model to start from its pre-trained state and gradually incorporate stylistic features during training, preventing sudden noise injection.
  • Conditioning: The model uses Canny Edge detection to provide a structural "skeleton" and Text Prompts (e.g., "Yao_Wen_Han style") to guide the aesthetic output.

Data and Training

The study utilized a curated dataset of 112 images, including 44 high-quality Jiehua paintings by masters like Zhang Xiao Gang and Yao Wen Han. Each image was paired with its Canny edge map and a descriptive text prompt. To enhance robustness, 50% of text prompts were dropped during training (Classifier-Free Guidance), forcing the model to rely more heavily on the visual edge conditions.

Results and Discussion

The FSDMC model demonstrated a significant performance advantage over CycleGAN across both quantitative and qualitative metrics.

Quantitative Performance

The Fréchet Inception Distance (FID) score for FSDMC was 3.27, compared to 56 for CycleGAN. This indicates that the images generated by FSDMC are statistically much closer to the distribution of real Jiehua paintings.

Qualitative Analysis

Expert evaluations highlighted the model's success in applying specific artist styles while maintaining structural integrity.

Qualitative results of the FSDMC model, demonstrating the transformation from original images to Jiehua style using Canny edge control conditions and text prompts.

As illustrated in the inference examples, the model successfully transforms modern or general landscape scenes into the Jiehua style. The use of Canny edges prevents the "melting" of architectural lines, a common failure mode in GAN-based approaches. The resulting images exhibit the characteristic ink textures and shading of the target style without losing the semantic content of the original scene.

Conclusion

The FSDMC approach proves highly effective for specialized style transfer tasks with limited data. By building upon a large pre-trained foundation and adding precise spatial controls, the model captures the unique geometric and textural requirements of Jiehua. Future work will focus on expanding the dataset and exploring the model's applicability to other traditional Chinese art forms, such as fluid ink-wash painting.

Summary updated

BibTeX
@misc{gu2024jiehua,
  author = {Yujia Gu and Haofeng Li and Xinyu Fang and Zihan Peng and Yinan Peng},
  title = {{JieHua Paintings Style Feature Extracting Model using Stable Diffusion with ControlNet}},
  year = {2024},
  eprint = {2408.11744},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.11744},
}
Figure