MetaHTR: Towards Writer-Adaptive Handwritten Text Recognition

Ayan Kumar Bhunia, Shuvozit Ghose, Amandeep Kumar, Pinaki Nath Chowdhury, Aneeshan Sain, Yi-Zhe Song

Abstract

Handwritten Text Recognition (HTR) remains a challenging problem due to the vast diversity in individual writing styles. Traditional approaches often assume that a limited number of styles can be captured by large-scale datasets. In this paper, we propose a different perspective: there is always a new, drastically different style, and only limited data will be available for adaptation during inference. We introduce MetaHTR, a novel meta-learning framework that exploits a small support set to output a writer-adapted model via a single gradient step update. By leveraging the insight that specific characters often exhibit larger style discrepancies, we propose meta-learning instance-specific weights for a character-wise cross-entropy loss. Our framework is model-agnostic and achieves an average performance gain of 5-7% with very few samples.

Introduction

Handwriting is deeply personal. While modern deep learning models for HTR perform well on average, they often struggle with the "long tail" of unique writing styles. Most existing models generalize by finding a "mean" style, failing when encountering a writer whose characters—like a cursive 'G' or an idiosyncratic 'Z'—deviate significantly from the training distribution.

Traditional solutions like fine-tuning are impractical for real-time applications as they require substantial data and numerous gradient steps. MetaHTR addresses this by "learning how to learn" a specific individual's handwriting on the fly. By employing a bi-level optimization strategy with character-wise importance weighting, the system achieves significant accuracy boosts during inference with only a single gradient update using as few as 16 word samples.

Methodology

MetaHTR is designed to be implemented on top of existing state-of-the-art architectures such as ASTER, SAR, or SCATTER. The core innovation lies in the application of Model-Agnostic Meta-Learning (MAML) tailored for sequential text data.

The MetaHTR framework featuring bi-level optimization and character-wise importance weighting via the $g_\gamma$ module.

Key Components:

  • Bi-level Optimization: During training, the model is exposed to "episodes" where it adapts to a specific writer using a small support set (inner loop) and is then evaluated on a validation set from the same writer (outer loop).
  • Character-Wise (CW) Learnable Weights: Not all characters are equally difficult to adapt. MetaHTR uses a small 3-layer MLP ($g_\gamma$) to predict an importance weight for each character's loss. This module analyzes the gradients of the current character relative to the mean gradient to identify characters representing the largest style discrepancies.
  • Layer-wise Learnable Learning Rates: Instead of a global learning rate, the model learns specific rates for every layer, allowing for nuanced updates during the fast adaptation phase.

Experimental Setup

The framework was evaluated on two benchmark datasets: IAM (English, 657 writers) and RIMES (French, 1,300 writers). Writers in the test set were entirely unseen during the meta-training phase to ensure true writer-adaptive performance.

Results and Discussion

MetaHTR consistently outperformed baselines across all models. For the ASTER baseline on the IAM dataset, Word Recognition Accuracy (WRA) increased from 81.3% to 89.2%. The model reaches peak performance with only 16 samples, proving its few-shot effectiveness.

Qualitative comparison of recognition results before and after writer adaptation, highlighting corrected character-level errors.

Key Insights:

  • Localized Discrepancy: Writers often have standard styles for most letters but unique styles for a few. MetaHTR’s learnable weights successfully prioritize these "outlier" characters during adaptation.
  • Gradients as Style Signatures: The gradient of the loss with respect to the classification layer contains rich information about style disagreement. By feeding these into the weight-prediction module, the model recognizes when it is "confused" by a specific style.

Conclusion

MetaHTR provides a commercially viable path toward personalized HTR. By enabling a "single-step" adaptation during inference, it allows devices to instantly tune recognition engines to a user's specific hand. Future work will explore adapting to specific vocabularies and linguistic quirks alongside visual style, further closing the gap between generic models and human-like recognition.

Summary updated

Abstract

Handwritten Text Recognition (HTR) remains a challenging problem to date, largely due to the varying writing styles that exist amongst us. Prior works however generally operate with the assumption that there is a limited number of styles, most of which have already been captured by existing datasets. In this paper, we take a completely different perspective -- we work on the assumption that there is always a new style that is drastically different, and that we will only have very limited data during testing to perform adaptation. This results in creates a commercially viable solution -- being exposed to the new style, the model has the best shot at adaptation, and the few-sample nature makes it practical to implement. We achieve this via a novel meta-learning framework which exploits additional new-writer data via a support set, and outputs a writer-adapted model via single gradient step update, all during inference. We discover and leverage on the important insight that there exists few key characters per writer that exhibit relatively larger style discrepancies. For that, we additionally propose to meta-learn instance specific weights for a character-wise cross-entropy loss, which is specifically designed to work with the sequential nature of text data. Our writer-adaptive MetaHTR framework can be easily implemented on the top of most state-of-the-art HTR models. Experiments show an average performance gain of 5-7% can be obtained by observing very few new style data.
BibTeX
@inproceedings{bhunia2021metahtr,
  author = {Ayan Kumar Bhunia and Shuvozit Ghose and Amandeep Kumar and Pinaki Nath Chowdhury and Aneeshan Sain and Yi-Zhe Song},
  title = {{MetaHTR: Towards Writer-Adaptive Handwritten Text Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=AZLFlLZjWE},
}
Figure