A Multi-Style Chinese Characters Writing Intelligent Tool Based on Small-scale Training Data

Zhen Zeng, Jie Wang, Xi LyuAAAI 2025 · DOI

TL;DR

This paper introduces an intelligent tool that allows users to create personalized Chinese character art in real-time using their own handwriting. By leveraging diffusion models and LoRA (Low-Rank Adaptation) techniques trained on small, curated datasets, the tool bridges the gap between traditional art and modern AI generation, making Chinese calligraphy more accessible and engaging for a wider audience.

What Problem Does This Paper Address?

Chinese characters are not just a writing system but also a rich art form with diverse historical and stylistic variations. However, mastering these styles requires extensive practice, limiting public participation. While existing AI tools can generate Chinese character art, they often focus on text-to-image generation or lack the nuanced control that reflects the actual writing process. This disconnect hinders genuine engagement and the appreciation of the art form. The authors aim to solve this by creating a tool that empowers non-professionals, including non-native speakers, to create personalized Chinese character art through an intuitive, handwriting-based interface that captures authentic stylistic elements.

Methods

The core of the tool is a diffusion model augmented with several LoRA models. LoRA is a parameter-efficient fine-tuning technique that allows for adapting large pre-trained models to specific tasks or styles with significantly less data and computational resources compared to full fine-tuning.

The process involves:

  1. Handwriting Input: Users provide their input via handwriting.
  2. Image Generation: This handwriting is converted into a stylized image using a ComfyUI workflow that incorporates img2img (image-to-image translation) and ControlNet.
  3. Style Adaptation: LoRA models, trained on small, curated datasets of specific Chinese character art styles, are applied to imbue the generated image with the desired artistic essence.
  4. Parameter Optimization: The authors used an X/Y/Z Plot script to systematically determine optimal LoRA model parameters and intensity settings for each style, ensuring the best possible generation. They also manually adjusted tags for training data to improve accuracy, as standard tags were often insufficient for traditional art.

Datasets

The authors created their own datasets by scanning and processing approximately 50 high-resolution images per art style category. These images were sourced from old books and collections. They utilized the WD14-tagger for initial tagging, but then manually refined these tags to ensure accurate training of the LoRA models, especially for styles where traditional art was underrepresented in general diffusion model training data.

Key Insights and Ideas

  • LoRA for Artistic Styles: LoRA models are highly effective for capturing diverse artistic styles of Chinese characters, even when trained on relatively small datasets. This makes it feasible to create a multi-style tool without requiring massive computational resources.
  • Handwriting as the Core Interaction: Emphasizing handwriting as the primary input method is crucial for fostering public engagement and preserving the authentic experience of Chinese character art creation, moving beyond simple text-to-image prompts.
  • Manual Tagging is Essential for Art: Standard image tagging tools are often insufficient for nuanced artistic styles. Manual curation and adjustment of tags are vital for accurate LoRA model training in specialized domains like traditional Chinese art.
  • Bridging Art and AI: The tool successfully merges the artistic tradition of Chinese calligraphy with modern generative AI, making it accessible to a broader audience and encouraging creative exploration.

Key Results

The paper highlights qualitative results demonstrating the tool's ability to generate visually striking Chinese character artworks that faithfully replicate user handwriting while adopting diverse artistic styles. Comparative assessments showed that their approach, using diffusion models with LoRA, significantly outperformed general AIGC tools like Midjourney and DALL-E, as well as a standard diffusion img2img workflow without LoRA, in accurately capturing the unique stylistic essence of traditional Chinese character art. User feedback from an exhibition indicated high satisfaction, with participants rating their creations 4-5 out of 5 and expressing increased interest in Chinese character writing.

Limitations and Future Work

The authors acknowledge that the time required for dataset creation and training currently limits the diversity of styles available in their tool. They suggest that future work will focus on expanding style diversity to better reflect the aesthetic variety of Chinese character art and further promote broader public participation in this art form.

Summary updated

BibTeX
@inproceedings{zeng2025multi,
  author = {Zhen Zeng and Jie Wang and Xi Lyu},
  title = {{A Multi-Style Chinese Characters Writing Intelligent Tool Based on Small-scale Training Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35381},
}

Loading related content...

Figure