A New Chinese Landscape Paintings Generation Model based on Stable Diffusion using DreamBooth

Yujia Gu, Xinyu Fang, Xueyuan Deng, Zihan Peng, Yinan PengarXiv 2024 · DOI · arXiv

Introduction

Traditional Chinese landscape painting is defined by unique brushwork, spatial compositions, and "spirit resonance"—qualities that general-purpose text-to-image models often struggle to replicate. While models like Stable Diffusion are powerful, they are typically "data-hungry" and fail to capture the nuances of specific artistic niches without specialized training. This study explores parameter-efficient fine-tuning (PEFT) methods to adapt Stable Diffusion for high-fidelity Chinese landscape painting generation.

The Diffusion Framework

The foundation of this work is the diffusion model, which operates through two primary phases: a forward process that incrementally adds Gaussian noise to an image until it becomes pure noise, and a reverse process where a neural network (typically a U-Net) learns to remove that noise to recover the original data distribution.

Overview of the forward and reverse diffusion processes used in generative modeling.

By training on a curated dataset of 1,973 Chinese landscape paintings and 6,335 modern landscapes, the researchers aimed to guide this reverse process toward the specific aesthetic of ink and wash paintings.

Fine-Tuning with DreamBooth and LoRA

The researchers compared two popular PEFT strategies: Low-Rank Adaptation (LoRA) and DreamBooth. While LoRA is computationally efficient by updating only a small fraction of parameters, DreamBooth proved more effective for capturing complex artistic styles.

DreamBooth, originally intended for subject-driven generation, was adapted here to treat "Chinese Landscape Painting" as a unique subject. By associating the style with a unique identifier (e.g., [V]) and a class name, the model learns the structural essence of the art form. A critical component is the class-specific prior preservation loss, which prevents the model from collapsing into repetitive outputs and maintains diversity.

The DreamBooth fine-tuning pipeline, mapping input images and class names to a personalized text-to-image model.

Evaluation and Results

The performance was measured using the Fréchet Inception Distance (FID) and qualitative assessments from six art experts. DreamBooth significantly outperformed both the base Stable Diffusion model and the LoRA-tuned version.

Model FID Score (Lower is Better)
Pre-trained Stable Diffusion (Base) 26.78
LoRA Fine-tuned SDM 17.85
DreamBooth Fine-tuned SDM 12.75

Expert evaluations confirmed these findings, with the DreamBooth model receiving the highest marks for authenticity and stylistic "fit." This alignment between mathematical metrics and human aesthetic judgment underscores the model's success in capturing the "spirit" of the medium.

Conclusion

This research demonstrates that specialized fine-tuning on a relatively small, high-quality dataset can yield state-of-the-art results in niche artistic domains. While the current model is highly specialized, future work will focus on improving style generalization across different historical periods and increasing output resolution to better capture fine-grained ink textures.

Summary updated

BibTeX
@misc{gu2024new,
  author = {Yujia Gu and Xinyu Fang and Xueyuan Deng and Zihan Peng and Yinan Peng},
  title = {{A New Chinese Landscape Paintings Generation Model based on Stable Diffusion using DreamBooth}},
  year = {2024},
  eprint = {2408.08561},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.08561},
}
Figure