TL;DR
DRA-HTR is an unsupervised domain adaptation pipeline that allows Handwritten Text Recognition (HTR) models to learn new handwriting styles and document formats without requiring any new manual transcriptions. By disentangling textual content from visual style and leveraging Large Language Models (LLMs) for post-recognition correction, the framework achieves state-of-the-art performance across diverse historical and modern datasets.
What Problem Does This Paper Address?
Handwritten Text Recognition (HTR) has improved significantly with deep learning, but it remains notoriously brittle. A model trained on modern English letters will likely fail when faced with 18th-century German manuscripts due to "domain shift"—differences in handwriting style, ink degradation, paper texture, and linguistic patterns.
While researchers often have access to large volumes of unlabeled document images from new domains, manually transcribing them to retrain a model is prohibitively expensive and slow. The authors aim to bridge this gap using Unsupervised Domain Adaptation (UDA): teaching a model to recognize text in a new "target" domain by using labeled data from a "source" domain and only unlabeled images from the target.
Methods
The authors propose DRA-HTR (Disentangling Representation and Adaptation for HTR), which builds upon the earlier DRANet framework. The process follows a four-stage pipeline:
- Source Pretraining: An HTR model (a CRNN) is trained on labeled source data.
- Visual Domain Adaptation: A generative network learns to separate images into content (the letters) and style (the visual appearance). It then recombines source content with target style to create "synthetic" training samples that look like the target domain but retain the source labels.
- Fine-Tuning: The HTR model is fine-tuned on these style-transferred images.
- Language Adaptation (DRAL-HTR): The authors introduce a post-processing step using the Qwen LLM to refine the HTR output, using neighboring lines and multiple recognition hypotheses as context.
Key Architectural Innovations:
- ConvNeXt-V2 Encoder: Replaces simpler backbones to better capture multiscale features of complex handwriting.
- FiLM (Feature-wise Linear Modulation): Used in the generator to precisely inject target style into the source content features at multiple stages.
- PatchGAN Discriminator: Instead of judging an entire line as "real" or "fake," this evaluates local patches, which is better for capturing granular details like stroke width and ink texture.
- Progressive Training: A strategy that delays the introduction of recognition-based losses until the generator has learned a basic style transfer, preventing noisy gradients from destabilizing the model.
Datasets
The framework was evaluated across seven HTR datasets spanning four languages:
- English: IAM (modern) and Bentham (historical).
- German: READ 2016 (historical).
- Norwegian: NorHand (two distinct partitions).
- Swedish: Riksarkivet (two distinct partitions).
These datasets provide a rigorous test of the model's ability to handle variations in language, historical period, and document degradation.
Key Insights and Ideas
- Disentanglement is Key: By explicitly separating style from content, the model can generate training data that is visually indistinguishable from the target domain while maintaining 100% accurate labels.
- Local Realism Matters: The switch to a PatchGAN discriminator highlights that for HTR, "realism" is found in the local textures of ink and paper rather than the global structure of the page.
- Training Stability: One of the biggest contributions is the massive reduction in training variance. Previous generative DA methods were often "hit or miss"; DRA-HTR produces consistent results across independent runs.
- LLMs as Contextual Correctors: Using an LLM to choose between the top 20 "hypotheses" from the HTR model (beam search) proved more effective than simple greedy decoding, especially when the LLM was allowed to see the preceding and following lines.
Key Results
- Superior Accuracy: DRA-HTR reduced the average Character Error Rate (CER) by 13.52 percentage points compared to the previous state-of-the-art domain adaptation method.
- Robustness: The average standard deviation across training runs decreased from 7.23 to 1.11 percentage points, indicating a much more stable training process.
- LLM Boost: The addition of LLM-based post-correction (DRAL-HTR) further reduced the CER by an average of 2.16 percentage points.
- Zero-Shot vs. Adapted: In one example (IAM to Bentham), a standard model had a 13.23% CER; after DRA-HTR adaptation, this dropped to 5.29%, and further to 3.63% with LLM correction.
Limitations and Future Work
- Computational Cost: Using a large LLM like Qwen for post-correction is computationally expensive and significantly slower than the HTR model's native inference.
- Language Bias: The LLM improvements were most pronounced in English. The authors noted that gains were more limited for other languages, likely due to the LLM's training distribution or the specific historical dialects involved.
- Future Directions: The authors suggest exploring more efficient language adaptation strategies that don't rely on massive LLMs and further optimizing hyperparameters for specific language-target pairs.
Summary updated