Text and Style Conditioned GAN for Generation of Offline Handwriting Lines

Brian L. Davis, Chris Tensmeyer, Brian L. Price, Curtis Wigington, Bryan S. Morse, Rajiv Jain

TL;DR

This paper introduces a novel Generative Adversarial Network (GAN) capable of generating realistic, arbitrary-length lines of handwritten text conditioned on both the desired text content and a specific writing style. This advancement moves beyond previous methods that focused on stroke generation or single words, offering a more complete and versatile solution for synthesizing human-like handwriting.

What Problem Does This Paper Address?

The generation of realistic handwritten text has been a long-standing challenge. Prior work often focused on generating stroke points (requiring online handwriting data) or individual words, which limited their applicability. A key gap was the ability to generate entire lines of text with variable lengths, mimicking a specific author's style, using readily available offline handwriting images. This capability is valuable for applications like personalized digital assistants, assistive technologies for individuals with disabilities, and generating synthetic data for training handwriting recognition models.

Methods

The authors propose a GAN-based framework that combines several key components:

  • Generator (G): This network takes the desired text and a style vector as input to produce a handwritten line image. It's built upon the StyleGAN architecture, incorporating a novel approach to handle variable-length text by predicting character spacing.
  • Style Extractor (S): This component learns to extract a latent style vector from an example handwriting image. It uses a combination of convolutional layers and character-specific heads to capture stylistic nuances.
  • Spacing Network (C): Crucial for variable-length generation, this network predicts the horizontal layout of characters, including blanks and character repetitions, based on the text and style vector. This allows the model to generate lines of appropriate width.
  • Handwriting Recognition Network (R): A pre-trained network is used to enforce legibility and content accuracy in the generated text via a Connectionist Temporal Classification (CTC) loss.
  • Encoder (E): Used for perceptual loss, this network extracts features to ensure visual similarity between generated and real images.
  • Discriminator (D): A patch-based discriminator is employed to distinguish between real and generated handwriting images, driving the realism of the output.

The training process cleverly integrates GAN and autoencoder techniques, along with reconstruction and recognition losses, and employs a refined gradient balancing method to manage the different loss magnitudes.

Datasets

The primary datasets used in this work are:

  • IAM Handwriting Dataset: Used for training and validation, containing segmented handwriting words and lines with transcriptions.
  • RIMES Dataset: Used for evaluation and comparison with prior methods, also containing segmented handwriting words and lines.

The authors note that authors are disjoint across train/val/test splits for the IAM dataset.

Key Insights and Ideas

  • Variable-Length Generation: The introduction of a dedicated spacing network (C) is a significant contribution, enabling the generation of lines with widths that naturally adapt to the text content and writing style, rather than relying on fixed-size outputs or heuristics.
  • Style Extraction from Offline Images: The ability to extract meaningful style vectors from static images, rather than requiring online writing data, makes the approach more practical and widely applicable.
  • Legibility through Recognition Supervision: Leveraging a pre-trained handwriting recognition network (R) as a loss function is an effective way to ensure the generated text is legible and accurate, even without explicit character-level supervision during generation.
  • Human-like Realism: The combination of GANs, autoencoders, and carefully balanced losses results in generated images that are highly convincing to human evaluators, often fooling them into believing they are real handwriting.
  • Clustering of Styles: The style extractor (S) learns a latent space where styles from the same author naturally cluster together, even without explicit supervision for this behavior, suggesting the learned style representations are semantically meaningful.

Key Results

  • Human Evaluation: A user study showed that participants could only correctly identify generated images as computer-generated 52.2% of the time, indicating high realism. For deliberately poorly generated images, accuracy was still 89.5%, suggesting the model's outputs are not easily dismissed as artifacts.
  • Quantitative Scores: The model achieved competitive FID and GS scores on the RIMES dataset, outperforming some prior methods, particularly when trained on full lines rather than just words.
  • Qualitative Results: The generated images exhibit high visual quality, capturing global style elements like slant and size. Interpolations between styles also produce realistic intermediate handwriting.

Limitations and Future Work

  • Character Shape Mimicry: While the model captures global style, it sometimes struggles to perfectly replicate fine-grained character shape styles (e.g., the specific way a letter 'l' is looped).
  • Ethical Concerns: The authors acknowledge potential ethical concerns regarding the misuse of this technology for forgery, though they consider this minor as the method is not targeted at signatures and produces digital images.
  • Future Work: The paper suggests further investigation into the validity of FID for handwriting images and exploring more sophisticated methods for style extraction and generation. The ability to extract styles from images and generate arbitrary text opens doors for further personalization and creative applications.

Summary updated

Abstract

This paper presents a GAN for generating images of handwritten lines conditioned on arbitrary text and latent style vectors. Unlike prior work, which produce stroke points or single-word images, this model generates entire lines of offline handwriting. The model produces variable-sized images by using style vectors to determine character widths. A generator network is trained with GAN and autoencoder techniques to learn style, and uses a pre-trained handwriting recognition network to induce legibility. A study using human evaluators demonstrates that the model produces images that appear to be written by a human. After training, the encoder network can extract a style vector from an image, allowing images in a similar style to be generated, but with arbitrary text.
BibTeX
@inproceedings{davis2020text,
  author = {Brian L. Davis and Chris Tensmeyer and Brian L. Price and Curtis Wigington and Bryan S. Morse and Rajiv Jain},
  title = {{Text and Style Conditioned GAN for Generation of Offline Handwriting Lines}},
  booktitle = {BMVC},
  year = {2020},
  url = {https://openreview.net/forum?id=oWLXw6xINbs},
}
Figure