Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models

Bin Fu, Fanghua Yu, Anran Liu, Zixuan Wang, Jie Wen, Junjun He, Yu Qiao

Abstract

Few-shot font generation (FFG) for glyph-rich scripts like Chinese is a labor-intensive task that current generative models struggle to automate without structural artifacts. We propose MSD-Font, a framework that rethinks FFG by mimicking the systematic workflow of human designers. By decomposing the generation process into three distinct stages—Structure Construction, Font Transfer, and Font Refinement—within a Latent Diffusion Model (LDM) framework, we achieve high-fidelity font synthesis. Our approach utilizes a dual-network architecture to specialize in style adaptation and detail polishing, significantly outperforming existing GAN-based state-of-the-art methods in both quantitative metrics and visual quality.

Introduction

Few-shot font generation (FFG) aims to create a complete stylized font set from a limited number of reference images. This is particularly critical for scripts like Chinese, which contain thousands of unique characters. Traditional methods primarily rely on Generative Adversarial Networks (GANs) using a style-content disentanglement paradigm. However, these "all-at-once" approaches often fail to balance complex global structures with intricate local styles, leading to broken characters or inconsistent aesthetics. MSD-Font addresses these limitations by adopting a multi-stage diffusion process that mirrors the step-by-step refinement used by professional designers.

Methodology

MSD-Font is built upon Latent Diffusion Models (LDM), where the reverse diffusion process is strategically decomposed into three stages:

  1. Structure Construction Stage: The model generates the global structure of the target character using a source template as an anchor. This ensures topological correctness from the start.
  2. Font Transfer Stage: Within the time interval $(t_1, t_2]$, the model transforms the latent features of the source font into the target style. This is treated as a domain mapping task.
  3. Font Refinement Stage: In the final steps $(0, t_1]$, the model focuses on recovering fine-grained local details and polishing the final appearance.
The MSD-Font architecture (c) and the latent feature transition process (a, b) compared to traditional models.

To effectively manage these distinct tasks, the authors implement a dual-network approach. A single network often fails to master both style transfer and detail refinement simultaneously. By employing two specialized prediction networks ($z_{ heta1}$ and $z_{ heta2}$), the model achieves higher precision in both global style adaptation and local stroke recovery.

Experiments and Results

The model was evaluated using a dataset of 900 Chinese fonts. Training was conducted on 840 fonts, while testing focused on two challenging scenarios: Unseen Fonts, Unseen Contents (UFUC) and Unseen Fonts, Seen Contents (UFSC).

MSD-Font outperformed prominent state-of-the-art (SOTA) methods, including LF-Font, MX-Font, and FsFont. On the UFUC dataset, MSD-Font achieved a structural similarity (SSIM) of 0.7189 and a perceptual similarity (LPIPS) of 0.1527, surpassing its closest competitors.

Qualitative comparison of the generative process between traditional diffusion models and the multi-stage MSD-Font.

Qualitative analysis reveals that MSD-Font produces significantly cleaner strokes and more stable structures. While GAN-based methods frequently produce blurry or "broken" characters for complex glyphs, the multi-stage diffusion approach maintains character integrity while accurately applying the target style.

Conclusion

MSD-Font demonstrates that incorporating a structured, multi-stage process into diffusion models can solve the long-standing issues of structural instability in few-shot font generation. While the iterative nature of diffusion models introduces higher computational costs compared to GANs, the gains in visual fidelity and stylistic consistency are substantial. Future work will focus on optimizing inference speed and exploring cross-lingual applications.

Summary updated

Abstract

Few-shot font generation (FFG) produces stylized font images with a limited number of reference samples, which can significantly reduce labor costs in manual font designs. Most existing FFG methods follow the style-content dis-entanglement paradigm and employ the Generative Adver-sarial Network (GAN) to generate target fonts by combining the decoupled content and style representations. The complicated structure and detailed style are simultaneously generated in those methods, which may be the sub-optimal solutions for FFG task. Inspired by most manual font design processes of expert designers, in this paper, we model font generation as a multi-stage generative process. Specifically, as the injected noise and the data distribution in diffusion models can be well-separated into different sub-spaces, we are able to incorporate the font transfer process into these models. Based on this observation, we generalize diffusion methods to modelfont generative process by separating the reverse diffusion process into three stages with different functions: The structure construction stage first generates the structure information for the target character based on the source image, and the font transfer stage subsequently transforms the source font to the target font. Finally, the font refinement stage enhances the appearances and local details of the target font images. Based on the above multi-stage generative process, we construct our font generation framework. named MSD-Font, with a dual-network approach to generate font images. The superior performance demonstrates the effectiveness of our model. The code is available at: https://github.com/fubinfbIMSD-Font.
BibTeX
@inproceedings{fu2024generate,
  author = {Bin Fu and Fanghua Yu and Anran Liu and Zixuan Wang and Jie Wen and Junjun He and Yu Qiao},
  title = {{Generate Like Experts: Multi-Stage Font Generation by Incorporating Font Transfer Process into Diffusion Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00658},
}
Figure