Abstract
Style transfer in traditional Chinese landscape painting, particularly the architectural genre known as Jiehua, presents unique challenges in balancing global artistic texture with local structural precision. This paper introduces a Fine-tuned Stable Diffusion Model with ControlNet (FSDMC) designed to extract and apply these intricate stylistic features. By utilizing Canny edge detection for spatial conditioning and specific text prompts, the proposed model achieves a Fréchet Inception Distance (FID) of 3.27, vastly outperforming the industry-standard CycleGAN. Qualitative evaluations by experts further confirm the model's ability to preserve semantic information while accurately rendering the geometric hallmarks of Jiehua.
Introduction
Jiehua, or "boundary painting," is a sub-genre of Chinese painting characterized by the precise, ruler-assisted depiction of buildings and structures. Unlike more fluid ink-wash styles, Jiehua requires high geometric accuracy. Traditional style transfer methods, such as CycleGAN, often suffer from "catastrophic forgetting," where the semantic content of the original image is lost, or structural details are warped. There is a need for a generative approach that can capture the specific "fingerprints" of Jiehua artists—such as Yao Wen Han—without compromising the integrity of the scene's architecture.
Methodology
The researchers compared two primary architectures for this task: a baseline CycleGAN and the proposed FSDMC.
CycleGAN Baseline
CycleGAN is an unpaired image-to-image translation model utilizing dual generators and discriminators. It relies on cycle consistency loss to ensure that an image translated from Domain A to B can be mapped back to A. While effective for general texture transfer, it often fails to maintain the rigid structural lines essential to Jiehua.
FSDMC: Stable Diffusion with ControlNet
The proposed FSDMC leverages Stable Diffusion v1.5 augmented with ControlNet to provide fine-grained spatial control.
As shown in the architecture diagram above, the model employs several key mechanisms:
- Locked vs. Trainable Blocks: The pre-trained Stable Diffusion weights are "locked" to retain broad generative knowledge, while a "trainable copy" is fine-tuned on the Jiehua dataset.
- Zero Convolutions: These layers connect the ControlNet to the main model. Initialized at zero, they allow the model to start from its pre-trained state and gradually incorporate stylistic features during training, preventing sudden noise injection.
- Conditioning: The model uses Canny Edge detection to provide a structural "skeleton" and Text Prompts (e.g., "Yao_Wen_Han style") to guide the aesthetic output.
Data and Training
The study utilized a curated dataset of 112 images, including 44 high-quality Jiehua paintings by masters like Zhang Xiao Gang and Yao Wen Han. Each image was paired with its Canny edge map and a descriptive text prompt. To enhance robustness, 50% of text prompts were dropped during training (Classifier-Free Guidance), forcing the model to rely more heavily on the visual edge conditions.
Results and Discussion
The FSDMC model demonstrated a significant performance advantage over CycleGAN across both quantitative and qualitative metrics.
Quantitative Performance
The Fréchet Inception Distance (FID) score for FSDMC was 3.27, compared to 56 for CycleGAN. This indicates that the images generated by FSDMC are statistically much closer to the distribution of real Jiehua paintings.
Qualitative Analysis
Expert evaluations highlighted the model's success in applying specific artist styles while maintaining structural integrity.
As illustrated in the inference examples, the model successfully transforms modern or general landscape scenes into the Jiehua style. The use of Canny edges prevents the "melting" of architectural lines, a common failure mode in GAN-based approaches. The resulting images exhibit the characteristic ink textures and shading of the target style without losing the semantic content of the original scene.
Conclusion
The FSDMC approach proves highly effective for specialized style transfer tasks with limited data. By building upon a large pre-trained foundation and adding precise spatial controls, the model captures the unique geometric and textural requirements of Jiehua. Future work will focus on expanding the dataset and exploring the model's applicability to other traditional Chinese art forms, such as fluid ink-wash painting.
Summary updated