TL;DR
This paper introduces DeepCalliFont, a novel approach for synthesizing Chinese calligraphy fonts from just a few examples. It tackles the challenge of connected strokes in calligraphy by integrating glyph images and writing trajectories using a dual-modality generative model, achieving state-of-the-art results.
What Problem Does This Paper Address?
Creating Chinese calligraphy fonts is a labor-intensive and complex task. While existing font synthesis methods can generate regular fonts, they struggle with the unique characteristics of calligraphy, particularly connected strokes and non-standard stroke orders. These calligraphy styles often violate the implicit assumptions of component and stroke consistency that traditional methods rely on. The paper aims to bridge this gap by developing a method that can synthesize high-quality calligraphy fonts even with limited training data, specifically addressing the challenge of connected strokes.
Methods
DeepCalliFont employs a dual-modality approach, leveraging both glyph images and writing trajectories. The core idea is to represent characters in these two modalities and learn a "coordinated representation" where features from both are mapped to a common domain.
Key components include:
- Dual-Modality Generative Model: The model consists of two parallel branches: one for image synthesis and one for sequence (writing trajectory) generation.
- Image Feature Recombination (IFR) Module: This module uses attention mechanisms to fuse image and sequence features, allowing the sequence modality to guide the image synthesis and improve stroke details.
- Differentiable Rasterizer Loss: This novel loss function ensures that the synthesized writing trajectories are accurately placed within the target glyph image, crucial for connected strokes.
- Distillation-Restoration Process: This technique maps features from the image and sequence branches into a shared latent space using InfoNCE loss, promoting consistency.
- Two-Phase Pre-training Strategy: To overcome the scarcity of dual-modality data, the model is first pre-trained on large uni-modality datasets (images and trajectories separately) and then fine-tuned on a smaller dual-modality dataset.
Datasets
- Training Data:
- 251 fonts and the CASIA Online Chinese Handwriting Databases were used for the initial uni-modality pre-training of individual branches.
- For the dual-modality fine-tuning, 42 fonts from Jiang et al. (2019) were used, each comprising 3,000 glyph images and corresponding writing trajectories.
- Test Datasets:
- A regular font test set consisting of 30 fonts from Liu and Lian (2023).
- A calligraphy font test set comprising 10 calligraphy fonts.
Key Insights and Ideas
- Connected Strokes are the Bottleneck: The paper correctly identifies that the primary challenge in calligraphy font synthesis lies in handling connected strokes, which breaks traditional assumptions about character structure.
- Trajectories as a Complementary Modality: Writing trajectories offer a more direct representation of stroke order and connections than static glyph images, making them a powerful complementary modality for calligraphy synthesis.
- Attention for Cross-Modal Guidance: The IFR module's use of attention to let sequence features guide image generation is a key insight for improving fine-grained stroke details in calligraphy.
- Pre-training is Crucial: The two-stage pre-training strategy is essential for leveraging abundant uni-modality data to compensate for the limited availability of dual-modality data, significantly boosting performance.
Key Results
- State-of-the-Art Performance: DeepCalliFont outperforms several existing methods on calligraphy font synthesis across metrics like MAE, FID, and LPIPS. For instance, it achieves an MAE of 0.1254 and FID of 14.6 on calligraphy fonts, significantly better than competitors.
- Superiority in Connected Strokes: Qualitative comparisons (Figures 5 and 8) clearly show DeepCalliFont's ability to synthesize high-quality glyphs with connected strokes, a weakness of other methods.
- Effective Module Contributions: Ablation studies (Table 1) demonstrate that each proposed module (IFR, differentiable rasterizer loss, distillation-restoration) contributes to improved performance.
- Satisfactory Regular Font Synthesis: The method also performs well on regular fonts, showing its generalizability.
Limitations and Future Work
- Limitation: The paper acknowledges that few-shot font generation is inherently ill-posed, meaning synthesized images might not perfectly match the ground truth but should still be visually pleasing. More detailed discussions on failure cases are deferred to supplementary material.
- Future Work: Not explicitly detailed, but the success with dual-modality learning and the pre-training strategy suggests potential for further exploration in combining more modalities or scaling up the pre-training datasets. The focus on calligraphy also opens avenues for exploring other complex artistic font styles.
Summary updated