Abstract
In this paper, we demonstrate how a generative model can be used to build a better recognizer through the control of content and style. We are building an online handwriting recognizer from a modest amount of training samples. By training our controllable handwriting synthesizer on the same data, we can synthesize handwriting with previously underrepresented content (e.g., URLs and email addresses) and style (e.g., cursive and slanted). Moreover, we propose a framework to analyze a recognizer that is trained with a mixture of real and synthetic training data. We use the framework to optimize data synthesis and demonstrate significant improvement on handwriting recognition over a model trained on real data only. Overall, we achieve a 66% reduction in Character Error Rate.
Introduction: The Combinatorial Explosion of Handwriting
Training a robust handwriting recognizer is fundamentally a "combinatorial explosion" problem. Handwriting is defined by two independent axes: content (the characters written) and style (cursive vs. printed, neat vs. messy, slant, etc.).
Collecting a dataset that covers every possible style for every possible word is nearly impossible. Real-world datasets are often biased toward "typical" writing and common words, leaving recognizers struggling with rare styles or specialized text like URLs, hashtags, and email addresses. Standard data augmentation (like rotating or scaling images) fails to address this because it doesn't create truly new styles or semantic content.
Methodology: Controllable Generation and Data Incubation
The authors propose Data Incubation, a process that treats a generative model as a strategic data factory rather than just a random noise generator.
Controllable Generative Model
The system uses a sequence-to-sequence model that factorizes handwriting into content and style. This allows for the "rendering" of specific text strings in various styles—either mimicking a specific writer or sampling from a learned latent space. By forcing this separation, the model can "imagine" what a URL would look like in a messy cursive style, even if that specific combination was never seen during training.
Sampling Bias Control
A crucial technical lever in this method is the "sampling bias" ($b$). By adjusting the standard deviation of the model's output distribution, researchers can control the "neatness" of the output. High bias produces very neat, average-looking handwriting, while low bias produces more diverse, "messier" samples. Finding the "Goldilocks Zone"—where data is diverse enough to be useful but not so noisy as to be "garbage"—is essential for performance.
Recognition Framework
The synthetic data is evaluated using a recognizer consisting of a 2-layer CNN followed by a 2-layer bidirectional LSTM, trained with Connectionist Temporal Classification (CTC) loss.
Datasets and Evaluation
The study utilizes a proprietary dataset of 600,000 online handwriting samples (stylus movements) from 1,500 writers, covering English and five other European languages. To test the model's ability to handle complex, non-dictionary content, the researchers used a Special Patterns (SP) Set containing URLs, addresses, and emails.
Key Results
The impact of "incubating" data was significant across all test categories:
- English Recognition: Adding synthetic data with the same content as the training set reduced the error rate by 28%.
- Multilingual Recognition: On a standard test set, the framework achieved a 46% reduction in Character Error Rate (CER).
- Specialized Content: By expanding synthetic content to include "Special Patterns," the researchers achieved a massive 66% reduction in CER on both standard multilingual and specialized SP tests.
The synthetic data allowed the model to perform better on non-English languages despite having significantly less real-world training data for them.
Discussion and Limitations
The success of Data Incubation depends on the generative model's quality. If the generator introduces harmful artifacts (Case 1), it can degrade the recognizer. Furthermore, while the paper provides a framework for identifying the best sampling bias, finding the optimal setting still requires systematic experimentation. Future work may explore applying these techniques to offline handwriting (static images), which would require different generative architectures like GANs or Diffusion models to handle ink texture and background noise.
Summary updated