Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognition

Panagiotis Kaliosis, John PavlopoulosEMNLP 2025

TL;DR

This paper introduces FADA, a novel framework that tackles the challenge of character frequency distribution shifts in handwritten text recognition (HTR) and other sequence-to-sequence tasks. By aligning predicted character distributions with empirical ones during training and inference, FADA significantly improves transcription accuracy and robustness, especially for historical or domain-specific data.

What Problem Does This Paper Address?

Handwritten Text Recognition (HTR) systems aim to convert handwritten documents into machine-readable text. A significant challenge in this field is the inherent variability of handwriting, which can lead to shifts in character frequency distributions. These shifts occur due to linguistic evolution, orthographic reforms, or specific domain contexts (e.g., specialized jargon). Models trained on general datasets often underperform when applied to subsets with different character distributions, leading to systematic biases and reduced accuracy. Existing methods for improving robustness, like training on diverse corpora, don't always fully address these intra-dataset variations. This paper addresses the need for a method that explicitly accounts for and mitigates these character-level distributional shifts.

Methods

The core of the proposed solution is FADA (Frequency-Aware Distribution Alignment), which has two main components:

  • FADATR (Frequency-Aware Training): This is a novel training framework that incorporates an auxiliary loss function during training. This loss penalizes the divergence between the model's predicted character frequency distribution and an empirically derived target distribution for a given sample (e.g., based on its century or region). The Wasserstein distance (specifically, the second-order Wasserstein distance, W2) is used to quantify this divergence. This alignment loss is jointly optimized with the primary recognition loss (e.g., CTC or Cross-Entropy), allowing the model to learn general representations while adapting to subset-specific statistical patterns.
  • FADAGD (Frequency-Aware Guided Decoding): This component applies frequency alignment at inference time. It modifies the beam search decoding process by introducing a penalty term that scores candidate sequences based on how well their predicted character frequency distribution matches the empirical target distribution. This guides the search towards more accurate transcriptions without requiring model retraining, acting as a lightweight calibration mechanism.

Datasets

The authors evaluated FADA on several datasets:

  • HPGT Dataset (HTR): A Handwritten Paleographic Greek Text recognition dataset with images segmented into line-level data from the 10th to 16th centuries CE. This dataset exhibits clear intra-dataset character distribution shifts across centuries.
  • CATMUS-FR Dataset (HTR): A multilingual manuscript dataset, with the French subset used here, spanning four centuries (13th to 16th) and six distinct script types.
  • Synthetic Dataset (HTR): A custom dataset generated to exhibit controlled character frequency shifts across simulated "centuries" using text from four different Large Language Models (LLMs).
  • EdAcc Dataset (ASR): An Automatic Speech Recognition dataset with 40 hours of transcribed speech from speakers with diverse linguistic backgrounds and English accents.

Key Insights and Ideas

  • Character-level distribution shifts are a significant, often overlooked, problem: The paper highlights how subtle changes in character frequencies, driven by linguistic evolution or domain-specific language, can substantially degrade HTR model performance.
  • Alignment at both training and inference is powerful: FADA demonstrates that combining frequency-aware training (FADATR) with guided decoding (FADAGD) yields the best results, suggesting a synergistic effect.
  • Inference-time alignment is practical and effective: FADAGD offers a way to improve accuracy without retraining, making it valuable for low-resource scenarios or quick adaptation.
  • FADA is applicable beyond HTR: The lipogram generation experiment shows FADA's potential for controlling text generation by enforcing character-level constraints.
  • Frequency-aware training can reduce reliance on large beam sizes: The study shows that FADATR can achieve better results with smaller beam sizes compared to standard fine-tuning, indicating improved inherent model capabilities.

Key Results

  • Consistent performance gains: FADA (specifically FADATR-GD) consistently improved Character Error Rate (CER) and Word Error Rate (WER) across multiple HTR datasets (HPGT, CATMUS, Synthetic) and models (TrOCR, C-RNN), outperforming baselines. For example, on the HPGT dataset, FADATR-GD achieved CERs of 23.89% and 24.41% for TrOCR and C-RNN respectively, compared to baseline FT scores of 26.89% and 25.06%.
  • ASR improvements: FADA also showed benefits in ASR, with Whisper achieving a CER of 26.70% with FADATR-GD compared to 27.11% for standard FT on the EdAcc dataset.
  • Robustness to temporal/regional shifts: The per-century and per-region analysis showed that FADA models narrowed the performance gap between easier and more challenging subsets, demonstrating improved robustness.
  • Lipogram generation success: FADAGD significantly reduced the violation rate for forbidden characters (e.g., an 18.3% relative improvement for the letter 'E'), while only slightly impacting perplexity and readability.

Limitations and Future Work

  • Dependence on empirical distributions: FADA's effectiveness relies on the availability of well-defined empirical character frequency distributions. In highly variable or undefined scenarios, its impact might be reduced.
  • Unigram focus: The current approach focuses on unigram frequency alignment. Future work could explore higher-order linguistic patterns like bigrams and trigrams.
  • Computational overhead: While modest, FADA introduces a slight increase in training time (4-6%) and decoding time (7-9%). Optimizing efficiency for real-time applications is an area for future research.
  • Future directions: The authors suggest extending FADA to incorporate bi-grams and tri-grams, enabling sequence-level alignment, exploring case-sensitive distributions, and improving recognition of named entities and specialized terminology. Developing adaptive weighting strategies for the hyperparameter $\lambda$ is also proposed.

Summary updated

BibTeX
@inproceedings{kaliosis2025learning,
  author = {Panagiotis Kaliosis and John Pavlopoulos},
  title = {{Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure