Abstract
Handwritten Text Recognition (HTR) remains a challenging problem due to the vast diversity in individual writing styles. Traditional approaches often assume that a limited number of styles can be captured by large-scale datasets. In this paper, we propose a different perspective: there is always a new, drastically different style, and only limited data will be available for adaptation during inference. We introduce MetaHTR, a novel meta-learning framework that exploits a small support set to output a writer-adapted model via a single gradient step update. By leveraging the insight that specific characters often exhibit larger style discrepancies, we propose meta-learning instance-specific weights for a character-wise cross-entropy loss. Our framework is model-agnostic and achieves an average performance gain of 5-7% with very few samples.
Introduction
Handwriting is deeply personal. While modern deep learning models for HTR perform well on average, they often struggle with the "long tail" of unique writing styles. Most existing models generalize by finding a "mean" style, failing when encountering a writer whose characters—like a cursive 'G' or an idiosyncratic 'Z'—deviate significantly from the training distribution.
Traditional solutions like fine-tuning are impractical for real-time applications as they require substantial data and numerous gradient steps. MetaHTR addresses this by "learning how to learn" a specific individual's handwriting on the fly. By employing a bi-level optimization strategy with character-wise importance weighting, the system achieves significant accuracy boosts during inference with only a single gradient update using as few as 16 word samples.
Methodology
MetaHTR is designed to be implemented on top of existing state-of-the-art architectures such as ASTER, SAR, or SCATTER. The core innovation lies in the application of Model-Agnostic Meta-Learning (MAML) tailored for sequential text data.
Key Components:
- Bi-level Optimization: During training, the model is exposed to "episodes" where it adapts to a specific writer using a small support set (inner loop) and is then evaluated on a validation set from the same writer (outer loop).
- Character-Wise (CW) Learnable Weights: Not all characters are equally difficult to adapt. MetaHTR uses a small 3-layer MLP ($g_\gamma$) to predict an importance weight for each character's loss. This module analyzes the gradients of the current character relative to the mean gradient to identify characters representing the largest style discrepancies.
- Layer-wise Learnable Learning Rates: Instead of a global learning rate, the model learns specific rates for every layer, allowing for nuanced updates during the fast adaptation phase.
Experimental Setup
The framework was evaluated on two benchmark datasets: IAM (English, 657 writers) and RIMES (French, 1,300 writers). Writers in the test set were entirely unseen during the meta-training phase to ensure true writer-adaptive performance.
Results and Discussion
MetaHTR consistently outperformed baselines across all models. For the ASTER baseline on the IAM dataset, Word Recognition Accuracy (WRA) increased from 81.3% to 89.2%. The model reaches peak performance with only 16 samples, proving its few-shot effectiveness.
Key Insights:
- Localized Discrepancy: Writers often have standard styles for most letters but unique styles for a few. MetaHTR’s learnable weights successfully prioritize these "outlier" characters during adaptation.
- Gradients as Style Signatures: The gradient of the loss with respect to the classification layer contains rich information about style disagreement. By feeding these into the weight-prediction module, the model recognizes when it is "confused" by a specific style.
Conclusion
MetaHTR provides a commercially viable path toward personalized HTR. By enabling a "single-step" adaptation during inference, it allows devices to instantly tune recognition engines to a user's specific hand. Future work will explore adapting to specific vocabularies and linguistic quirks alongside visual style, further closing the gap between generic models and human-like recognition.
Summary updated