Data Incubation - Synthesizing Missing Data for Handwriting Recognition

Jen-Hao Rick Chang, Martin Bresler, Youssouf Chherawala, Adrien Delaye, Thomas Deselaers, Ryan S. Dixon, Oncel Tuzel

Abstract

In this paper, we demonstrate how a generative model can be used to build a better recognizer through the control of content and style. We are building an online handwriting recognizer from a modest amount of training samples. By training our controllable handwriting synthesizer on the same data, we can synthesize handwriting with previously underrepresented content (e.g., URLs and email addresses) and style (e.g., cursive and slanted). Moreover, we propose a framework to analyze a recognizer that is trained with a mixture of real and synthetic training data. We use the framework to optimize data synthesis and demonstrate significant improvement on handwriting recognition over a model trained on real data only. Overall, we achieve a 66% reduction in Character Error Rate.

Introduction: The Combinatorial Explosion of Handwriting

Training a robust handwriting recognizer is fundamentally a "combinatorial explosion" problem. Handwriting is defined by two independent axes: content (the characters written) and style (cursive vs. printed, neat vs. messy, slant, etc.).

Collecting a dataset that covers every possible style for every possible word is nearly impossible. Real-world datasets are often biased toward "typical" writing and common words, leaving recognizers struggling with rare styles or specialized text like URLs, hashtags, and email addresses. Standard data augmentation (like rotating or scaling images) fails to address this because it doesn't create truly new styles or semantic content.

Figure 1: The Data Incubation framework. (a) Real data is often sparse across the content-style matrix. (b-d) Synthetic data is used to fill missing content, styles, or both, addressing the 'missing gaps' (Case 3).

Methodology: Controllable Generation and Data Incubation

The authors propose Data Incubation, a process that treats a generative model as a strategic data factory rather than just a random noise generator.

Controllable Generative Model

The system uses a sequence-to-sequence model that factorizes handwriting into content and style. This allows for the "rendering" of specific text strings in various styles—either mimicking a specific writer or sampling from a learned latent space. By forcing this separation, the model can "imagine" what a URL would look like in a messy cursive style, even if that specific combination was never seen during training.

Figure 4: System architecture overview. (a) Online handwriting as stylus movements. (b) The controllable generative model factorizing style and content. (c) The recognizer architecture.

Sampling Bias Control

A crucial technical lever in this method is the "sampling bias" ($b$). By adjusting the standard deviation of the model's output distribution, researchers can control the "neatness" of the output. High bias produces very neat, average-looking handwriting, while low bias produces more diverse, "messier" samples. Finding the "Goldilocks Zone"—where data is diverse enough to be useful but not so noisy as to be "garbage"—is essential for performance.

Recognition Framework

The synthetic data is evaluated using a recognizer consisting of a 2-layer CNN followed by a 2-layer bidirectional LSTM, trained with Connectionist Temporal Classification (CTC) loss.

Datasets and Evaluation

The study utilizes a proprietary dataset of 600,000 online handwriting samples (stylus movements) from 1,500 writers, covering English and five other European languages. To test the model's ability to handle complex, non-dictionary content, the researchers used a Special Patterns (SP) Set containing URLs, addresses, and emails.

Key Results

The impact of "incubating" data was significant across all test categories:

  • English Recognition: Adding synthetic data with the same content as the training set reduced the error rate by 28%.
  • Multilingual Recognition: On a standard test set, the framework achieved a 46% reduction in Character Error Rate (CER).
  • Specialized Content: By expanding synthetic content to include "Special Patterns," the researchers achieved a massive 66% reduction in CER on both standard multilingual and specialized SP tests.

The synthetic data allowed the model to perform better on non-English languages despite having significantly less real-world training data for them.

Discussion and Limitations

The success of Data Incubation depends on the generative model's quality. If the generator introduces harmful artifacts (Case 1), it can degrade the recognizer. Furthermore, while the paper provides a framework for identifying the best sampling bias, finding the optimal setting still requires systematic experimentation. Future work may explore applying these techniques to offline handwriting (static images), which would require different generative architectures like GANs or Diffusion models to handle ink texture and background noise.

Summary updated

Abstract

In this paper, we demonstrate how a generative model can be used to build a better recognizer through the control of content and style. We are building an online handwriting recognizer from a modest amount of training samples. By training our controllable handwriting synthesizer on the same data, we can synthesize handwriting with previously underrepresented content (e.g., URLs and email addresses) and style (e.g., cursive and slanted). Moreover, we propose a framework to analyze a recognizer that is trained with a mixture of real and synthetic training data. We use the framework to optimize data synthesis and demonstrate significant improvement on handwriting recognition over a model trained on real data only. Overall, we achieve a 66% reduction in Character Error Rate.
BibTeX
@inproceedings{chang2022data,
  author = {Jen-Hao Rick Chang and Martin Bresler and Youssouf Chherawala and Adrien Delaye and Thomas Deselaers and Ryan S. Dixon and Oncel Tuzel},
  title = {{Data Incubation - Synthesizing Missing Data for Handwriting Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=iKO1AqiOOY},
}
Figure