Unsupervised Writer Adaptation for Synthetic-to-Real Handwritten Word Recognition

Lei Kang, Marcal Rusinol, Alicia Fornes, Pau Riba, Mauricio VillegasWACV 2020

TL;DR

This paper tackles the challenge of adapting handwritten text recognition (HTR) models to new writers without requiring any labeled data from those writers. They achieve this by training a model on synthetically generated text and then using an unsupervised domain adaptation technique to align the model's internal representations with real, unlabeled handwriting. This approach significantly improves recognition accuracy for new writers, making HTR more practical for real-world applications.

What Problem Does This Paper Address?

Handwritten Text Recognition (HTR) is notoriously difficult due to the immense variability in writing styles, both between different people (inter-writer variability) and even by the same person at different times (intra-writer variability). Training robust HTR models typically requires vast amounts of labeled data, which is expensive and time-consuming to collect. While synthetic data generation (using fonts to create artificial handwriting) can provide a large training corpus, models trained solely on synthetic data often struggle to generalize to real handwriting. This gap between synthetic and real data, known as the "domain shift," is a major hurdle. Existing adaptation methods often require labeled target data, which defeats the purpose of using synthetic data in the first place. This paper aims to bridge this gap by enabling unsupervised adaptation from synthetic to real handwriting, specifically for word recognition.

Methods

The core of the proposed method is an encoder-decoder architecture for handwritten word recognition, enhanced with an attention mechanism. This architecture is designed to handle variable-length input (word images) and output (character sequences).

The novelty lies in the unsupervised domain adaptation strategy integrated into the training process. This involves:

  • Synthetic Data Generation: A large corpus of synthetic word images is generated using a diverse set of electronic fonts and a substantial text corpus. This synthetic data is paired with its ground truth transcriptions.
  • Adversarial Training: The system is trained using an adversarial approach. A discriminator network is introduced to distinguish between features extracted from synthetic (labeled) and real (unlabeled) handwriting.
  • Gradient Reversal Layer (GRL): A GRL is used to reverse the gradient signal during backpropagation. This forces the encoder to learn feature representations that are indistinguishable to the discriminator, effectively aligning the synthetic and real data distributions. Simultaneously, the decoder is trained to perform accurate recognition on the aligned features.
  • Temporal Pooling: To handle the variable-length nature of handwriting sequences, several temporal pooling strategies (Column-wise Mean Value, Spatial Pyramid Pooling, Temporal Pyramid Pooling, and GRU) were explored. The Gated Recurrent Unit (GRU) was found to be the most effective for pooling variable-length encoder features into a fixed-size representation required by the discriminator.

Datasets

The authors utilized five publicly available datasets for their experiments:

  • GW: Historic English dataset.
  • IAM: Modern English dataset.
  • RIMES: Modern French dataset.
  • Esposalles: Historic Catalan dataset.
  • CVL: Modern English/German dataset.

They generated a new synthetic dataset by rendering over 5.6 million word images from a corpus of over 430,000 unique words across English, French, Catalan, and German languages, using 387 freely available electronic fonts. This synthetic dataset was augmented with various pixel-level and geometric transformations to mimic real handwriting variations.

Key Insights and Ideas

  • Synthetic data is a powerful pre-training tool: Even without any real data, a model trained on carefully generated synthetic data can achieve a reasonable baseline performance, demonstrating its potential for generalization.
  • Unsupervised adaptation is crucial for real-world HTR: The ability to adapt to new writers without manual annotation is a significant step towards making HTR practical for diverse, unconstrained scenarios.
  • Adversarial learning effectively bridges the domain gap: By forcing the model to produce features that are indistinguishable between synthetic and real data, the adversarial approach successfully aligns the feature distributions, leading to improved performance on real handwriting.
  • GRU-based temporal pooling is effective: The GRU proved to be the best strategy for condensing variable-length encoder outputs into a fixed representation for the discriminator, outperforming other pooling methods.
  • Adaptation benefits from more target data: The experiments showed that increasing the amount of unlabeled target data available for adaptation consistently leads to lower error rates.

Key Results

  • The proposed unsupervised writer adaptation method significantly reduces errors compared to a model trained solely on synthetic data.
  • Gap reduction (improvement from synthetic-only to adapted) ranged from 20% (CVL) to 60% (IAM).
  • On the IAM dataset, the adapted model achieved a CER of 13.58%, a substantial improvement over the synthetic-only baseline of 26.05%.
  • The method demonstrated competitive results across various datasets, including historic and modern documents, and single and multi-writer collections.
  • Even with a very limited number of samples (as few as 13 words per writer in the IAM dataset), the unsupervised adaptation still provided significant performance boosts.

Limitations and Future Work

  • Supervised vs. Unsupervised Performance: While the unsupervised approach offers significant advantages in terms of annotation cost, it does not reach the same performance levels as fully supervised fine-tuning methods that utilize labeled target data. The paper acknowledges this trade-off.
  • Visual Dissimilarity: For writers whose styles are visually very dissimilar to the synthetically generated material, the improvement from adaptation might be less pronounced.
  • Future Work: The authors suggest that this work is a promising direction towards a universal HTR for unconstrained scenarios, implying that further research could explore its application in more complex industrial settings. The flexibility of the approach, requiring only a few unlabeled images from the target domain, makes it well-suited for such applications.

Summary updated

BibTeX
@inproceedings{kang2020unsupervised,
  author = {Lei Kang and Marcal Rusinol and Alicia Fornes and Pau Riba and Mauricio Villegas},
  title = {{Unsupervised Writer Adaptation for Synthetic-to-Real Handwritten Word Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure