What is the Reward for Handwriting? -- Handwriting Generation by Imitation Learning

Keisuke Kanda, Brian Kenji Iwana, Seiichi UchidaarXiv 2020 · arXiv

TL;DR

Researchers at Kyushu University have applied Generative Adversarial Imitation Learning (GAIL) to model the human handwriting process. By treating handwriting as a reinforcement learning problem, the model learns to generate realistic trajectories with long-term "future planning" capabilities, successfully discovering the underlying "reward" of handwriting directly from data rather than relying on manually defined functions.

What Problem Does This Paper Address?

Handwriting generation is a complex task that requires more than just predicting the next point in a sequence. Human writers engage in future planning—for instance, when writing the digit '0', you move the pen with the intent of eventually closing the loop.

Traditional stochastic models (like HMMs) or simple supervised learning (like behavior cloning) are often "shortsighted" or Markovian, meaning they struggle with these long-term dependencies. While Reinforcement Learning (RL) is a natural fit for modeling goal-oriented behavior, it typically requires a manually defined reward function. Defining what makes a pen stroke "good" is incredibly difficult and prone to human bias. This paper asks: can we learn the reward function itself by observing expert handwriting?

Methods

The authors employ Generative Adversarial Imitation Learning (GAIL), a framework that combines RL with the adversarial training logic of GANs.

Training procedure of the GAIL-based model compared to typical RL

The system consists of three main modules, all implemented as 1D-Convolutional Neural Networks (1D-CNNs):

  1. The Actor (Generator): The "writer" that decides the next pen-tip position ($a_t$) based on the current partial trajectory ($s_t$).
  2. The Discriminator: Acts as the learned reward function. It is trained to distinguish between real human handwriting and the trajectories generated by the Actor.
  3. The Critic: Evaluates the "value" of a state, estimating the total expected future reward. This allows the Actor to make decisions based on long-term outcomes rather than just immediate feedback.

By training these modules together, the Actor learns to maximize the rewards provided by the Discriminator, effectively "imitating" the style and planning found in the training data.

Datasets

The models were trained and evaluated on two primary datasets:

  • UNIPEN 1a: An isolated digit dataset containing 11,078 samples. It includes various shapes like straight lines, curves, crossings, and closed loops.
  • UNIPEN 1b: An isolated alphabet dataset containing 15,661 samples, used to demonstrate the model's capability to acquire trends in arbitrary handwriting types.

Key Insights and Ideas

  • The Reward is Learned, Not Defined: The core "aha moment" is that the Discriminator successfully captures the essence of handwriting. Instead of a researcher guessing that "smoothness" or "speed" is the reward, the model discovers these features by trying to fool the discriminator.
  • Future Planning is Essential: The inclusion of a Critic module allows the model to estimate future rewards. This is what enables the model to generate well-balanced characters that occupy the intended space, a feat that simple prediction-based models often fail at.
  • Anisotropic Value Maps: By visualizing the $Q(s_t, a_t)$-maps (which show the expected reward for different next-moves), the authors found that the model's internal logic is highly flexible and non-Gaussian, adapting its "preferences" based on the current trajectory.
Visualization of learned Q-maps showing the probability of the next pen-tip position

Key Results

The GAIL-based model was compared against a standard CNN-based prediction model (supervised behavior cloning).

  • Spatial Awareness: The GAIL model generated trajectories that occupied the full image region more consistently. The prediction-based model often produced "shrunken" or incomplete characters.
  • Smoothness and Curvature: Quantitatively, the GAIL model's curvature distribution closely matched real human handwriting. Specifically, it was much better at generating the low-curvature (smooth) segments characteristic of natural writing, whereas the baseline model produced unnatural, jerky curves.
Qualitative comparison between the proposed GAIL model and a prediction-based baseline

Limitations and Future Work

While the model performs well on isolated characters, the authors note several areas for expansion:

  • Generalization: The current model is trained on specific datasets; future work aims to create a more general model by mixing various handwriting styles during training.
  • Discrimination Tasks: The learned reward function could potentially be used to distinguish between genuine handwriting and non-handwriting (or forgeries).
  • Medical Applications: A significant goal is applying this framework to medical diagnosis. Since conditions like Parkinson’s or Alzheimer’s affect handwriting trajectories, understanding the "reward" or "planning" changes in these patients could provide new diagnostic insights.

Summary updated

Abstract

Analyzing the handwriting generation process is an important issue and has been tackled by various generation models, such as kinematics based models and stochastic models. In this study, we use a reinforcement learning (RL) framework to realize handwriting generation with the careful future planning ability. In fact, the handwriting process of human beings is also supported by their future planning ability; for example, the ability is necessary to generate a closed trajectory like '0' because any shortsighted model, such as a Markovian model, cannot generate it. For the algorithm, we employ generative adversarial imitation learning (GAIL). Typical RL algorithms require the manual definition of the reward function, which is very crucial to control the generation process. In contrast, GAIL trains the reward function along with the other modules of the framework. In other words, through GAIL, we can understand the reward of the handwriting generation process from handwriting examples. Our experimental results qualitatively and quantitatively show that the learned reward catches the trends in handwriting generation and thus GAIL is well suited for the acquisition of handwriting behavior.
BibTeX
@misc{kanda2020what,
  author = {Keisuke Kanda and Brian Kenji Iwana and Seiichi Uchida},
  title = {{What is the Reward for Handwriting? -- Handwriting Generation by Imitation Learning}},
  year = {2020},
  eprint = {2009.10962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.10962},
}
Figure