DP-Font: Chinese Calligraphy Font Generation Using Diffusion Model and Physical Information Neural Network

Liguo Zhang, Yalong Zhu, Achref Benarab, Yusen Ma, Yuxin Dong, Jianguo Sun, Shengsheng Wang

This summary covers the paper "DP-Font: Chinese Calligraphy Font Generation Using Diffusion Model and Physical Information Neural Network," which presents a novel approach to digital calligraphy that blends generative AI with the laws of physics.

TL;DR

DP-Font is a generative framework that creates high-fidelity Chinese calligraphy by combining Denoising Diffusion Probabilistic Models (DDPM) with Physical Information Neural Networks (PINN). By explicitly encoding stroke order and incorporating the physical equations of ink diffusion into the learning process, the model produces characters that are structurally accurate and aesthetically "realistic" compared to traditional GAN-based methods.

What Problem Does This Paper Address?

Generating Chinese calligraphy is notoriously difficult for AI. Unlike standard fonts, calligraphy involves complex character shapes, personalized artistic styles, and the subtle "bleed" of ink on paper.

Existing methods generally fall into two camps:

  1. Graphics-based: These rely on manual stroke extraction, which is computationally expensive and often misses the "global" artistic style.
  2. Deep Learning-based (mostly GANs): While better at capturing style, Generative Adversarial Networks often suffer from unstable training, "collapsed" strokes, or a lack of physical realism.

The authors identify a key gap: most models treat characters as static images, ignoring the temporal sequence of writing (stroke order) and the physical behavior of ink.

Methods

The DP-Font architecture consists of two main pillars:

1. Multi-Attribute Conditional Diffusion

The model uses a diffusion backbone (DDPM) but conditions the generation on three specific attributes:

  • Content: The identity of the character.
  • Style: The specific "hand" or artistic flair of a calligrapher.
  • Stroke Order: This is a major contribution. The authors encode characters into a 36-dimensional vector based on five basic stroke types (horizontal, vertical, left-falling, dot, and turning). This provides a structural roadmap for the model, preventing the "nonsense" strokes often seen in AI-generated text.

2. Physical Information Neural Network (PINN)

To make the ink look real, the authors incorporate a physical diffusion equation directly into the neural network's loss function: $$\frac{\partial u}{\partial t} = D\nabla^2 u$$ Here, $u$ represents ink concentration. By adding a $L_{PINN}$ loss term, the model is forced to generate images that satisfy the laws of how fluid (ink) spreads over a medium (paper). This acts as a sophisticated regularizer, reducing over-smoothing and ensuring the edges of strokes look naturally "inked."

Datasets

  • Training Set: A compiled dataset of 100 different fonts, each containing 7,905 characters.
  • Source Material: Characters were sourced from the 2021 General Standard Chinese Character Stroke Order Standard to ensure authoritative structural data.
  • Test Set: 10 fonts and 200 intricate characters specifically chosen for their structural complexity.
  • Preprocessing: All characters were normalized to $80 \times 80$ pixels.

Key Insights and Ideas

  • Physics as a Constraint: The most "aha" moment is using the heat/diffusion equation to model ink. It moves the problem from pure "pixel-pushing" to "process-modeling."
  • Structural Guidance: By encoding stroke order, the model learns how a character is built, not just what it looks like. This significantly reduces structural errors (missing or redundant strokes) that plague other models.
  • Classifier-Free Guidance: The authors use a blending technique during inference to balance the influence of the style/stroke conditions against the unconditional model, allowing for precise control over the output's "calligraphic-ness."

Key Results

DP-Font was compared against five state-of-the-art models (FUNIT, SC-Font, MX-Font, DG-Font, and Diff-Font).

  • Quantitative: DP-Font achieved the best scores across all metrics:
  • SSIM (Structural Similarity): 0.735 (Highest)
    • FID (Frechet Inception Distance): 22.34 (Lowest, indicating the highest realism).
  • Improvement: It showed a notable 8.48% improvement in FID over the previous best diffusion-based method (Diff-Font).
  • Qualitative: Visual inspections show that DP-Font successfully imitates the styles of Tang Dynasty masters like Gongquan Liu and Zhenqing Yan, maintaining clear stroke boundaries and correct structural topology where other models produced "blurry" or broken characters.

Limitations and Future Work

  • Resolution: The current experiments are limited to $80 \times 80$ pixels. While sufficient for many applications, professional calligraphy often requires much higher resolutions to capture fine paper grain and "flying white" (dry brush) effects.
  • Computational Cost: Diffusion models are iteratively slow compared to GANs. While the paper mentions using DDIM to speed up sampling, real-time generation of large blocks of text remains a challenge.
  • Future Directions: The authors suggest that this physics-informed approach could be extended to other artistic mediums, such as oil painting or watercolor, where the physical interaction of the medium is central to the aesthetic.

Summary updated

BibTeX
@inproceedings{zhang2024dp,
  author = {Liguo Zhang and Yalong Zhu and Achref Benarab and Yusen Ma and Yuxin Dong and Jianguo Sun and Shengsheng Wang},
  title = {{DP-Font: Chinese Calligraphy Font Generation Using Diffusion Model and Physical Information Neural Network}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=SkrLT0tUq6},
}
Figure