Neural Style Difference Transfer and Its Application to Font Generation

Gantugs Atarsaikhan, Brian Kenji Iwana, Seiichi UchidaarXiv 2020 · arXiv

TL;DR

Researchers from Kyushu University have developed a novel method for automatic font generation that transfers the difference between two font styles onto a third target font. By introducing new "style difference" and "content difference" loss functions to the standard Neural Style Transfer (NST) framework, the system can perform complex edits like adding serifs or adjusting line weights without needing to redesign every character from scratch.

What Problem Does This Paper Address?

Designing a complete digital font is an arduous, professional-grade task. For complex writing systems like Japanese, which contains thousands of unique characters, designing each one individually is incredibly time-consuming.

While standard Neural Style Transfer (NST) can blend the style of one image into the content of another, it often lacks the precision needed for font design. A designer might not want to turn a font into a "Van Gogh painting"; they might simply want to make it "as bold as this other font" or "add the specific serifs from that font." This paper addresses the need for a more surgical approach: transferring a specific stylistic delta from one pair of fonts to a new target.

Methods

The authors propose Neural Style Difference Transfer. Unlike traditional NST, which uses two inputs (Content $C$ and Style $S$), this method uses three:

  1. Style Image 1 ($S_1$): A font with a specific feature (e.g., bold).
  2. Style Image 2 ($S_2$): The same font without that feature (e.g., regular).
  3. Content Image ($C$): The target font to be modified.

The goal is to generate an image $X$ such that the relationship between $X$ and $C$ mirrors the relationship between $S_1$ and $S_2$.

The core concept: transferring the difference between two 'H' fonts to a 'B' font.

The Loss Functions

The researchers modified the standard VGGNet-based optimization by introducing two new loss components:

  • Style Difference Loss: Instead of matching Gram matrices directly, the algorithm minimizes the Mean Squared Error (MSE) between the difference of Gram matrices of the style pair $(G_{S1} - G_{S2})$ and the difference of Gram matrices of the generated/content pair $(G_X - G_C)$.
  • Content Difference Loss: Similarly, it matches the difference in high-level feature maps between the style pair and the target pair. This captures structural differences, such as the presence or absence of serifs.

The entire process is visualized in their architecture overview: Overview of the VGGNet-based difference transfer architecture.

Overview of the VGGNet-based difference transfer architecture.

Datasets

The paper does not utilize a specific named large-scale dataset. Instead, the authors conducted experiments using various combinations of existing digital fonts to demonstrate specific capabilities:

  • Pairs of fonts from the same family (e.g., Serif vs. Sans-Serif versions).
  • Fonts with varying weights (Bold vs. Light).
  • Fonts with specific missing components to test "part transfer."

Key Insights and Ideas

  • Style Arithmetic: The fundamental insight is that font style can be treated as a vector operation. If you can define the "vector" that turns a Sans-Serif 'A' into a Serif 'A', you can apply that same vector to a Sans-Serif 'B'.
  • Layer Sensitivity: The authors found that different layers of the VGGNet capture different aspects of the "difference." Lower layers handle fine details, while higher layers handle global arrangements. They determined that using a weighted combination of all style layers (conv1_2 through conv5_2) was necessary to capture a complete stylistic difference.
  • Complementing vs. Erasing: The method is bi-directional. By swapping $S_1$ and $S_2$, the system can either add a feature (like serifs) or remove it (eroding the font).

Key Results

The authors demonstrated successful results across several font-editing tasks:

  • Serif Generation: Successfully added serifs to sans-serif fonts by using a serif/sans-serif pair as the style reference.
  • Line Width Transfer: The system could widen or narrow the strokes of a target font. However, they noted it struggled when the target font was already significantly wider than the reference style.
  • Missing Part Transfer: In cases where one style font had extra decorative elements, the system successfully "grafted" those elements onto the target font.
Results showing the transfer of missing parts and serifs.

Limitations and Future Work

Despite the promising results, the authors identified several hurdles:

  • Input Sensitivity: The target font ($C$) must be somewhat similar in style to the reference font ($S_2$) for the subtraction/addition logic to work effectively. If the styles are too disparate, the optimization produces artifacts.
  • Computational Cost: Because the method relies on iterative back-propagation (using L-BFGS or Adam) for every single stylization, it is relatively slow compared to feed-forward neural networks.
  • Character Alignment: A major failure case occurs when the characters in the style pair and the content image are different (e.g., trying to transfer the difference between 'A's to a 'W'). The structural mismatch can lead to incomplete or distorted fonts.

Future work suggests exploring encoder-decoder architectures or Generative Adversarial Networks (GANs) to speed up the process and improve robustness against character mismatches.

Summary updated

Abstract

Designing fonts requires a great deal of time and effort. It requires professional skills, such as sketching, vectorizing, and image editing. Additionally, each letter has to be designed individually. In this paper, we will introduce a method to create fonts automatically. In our proposed method, the difference of font styles between two different fonts is found and transferred to another font using neural style transfer. Neural style transfer is a method of stylizing the contents of an image with the styles of another image. We proposed a novel neural style difference and content difference loss for the neural style transfer. With these losses, new fonts can be generated by adding or removing font styles from a font. We provided experimental results with various combinations of input fonts and discussed limitations and future development for the proposed method.
BibTeX
@misc{atarsaikhan2020neural,
  author = {Gantugs Atarsaikhan and Brian Kenji Iwana and Seiichi Uchida},
  title = {{Neural Style Difference Transfer and Its Application to Font Generation}},
  year = {2020},
  eprint = {2001.07321},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.07321},
}

Loading related content...

Figure