DRA-HTR: Disentangling Representation and Adaptation for Handwritten Text Recognition

Gayan H. Pathirage, Florent Imbert, Denis Coquenet, Elisa H. Barney SmitharXiv 2026 · arXiv

TL;DR

DRA-HTR is an unsupervised domain adaptation pipeline that allows Handwritten Text Recognition (HTR) models to learn new handwriting styles and document formats without requiring any new manual transcriptions. By disentangling textual content from visual style and leveraging Large Language Models (LLMs) for post-recognition correction, the framework achieves state-of-the-art performance across diverse historical and modern datasets.

What Problem Does This Paper Address?

Handwritten Text Recognition (HTR) has improved significantly with deep learning, but it remains notoriously brittle. A model trained on modern English letters will likely fail when faced with 18th-century German manuscripts due to "domain shift"—differences in handwriting style, ink degradation, paper texture, and linguistic patterns.

While researchers often have access to large volumes of unlabeled document images from new domains, manually transcribing them to retrain a model is prohibitively expensive and slow. The authors aim to bridge this gap using Unsupervised Domain Adaptation (UDA): teaching a model to recognize text in a new "target" domain by using labeled data from a "source" domain and only unlabeled images from the target.

Methods

The authors propose DRA-HTR (Disentangling Representation and Adaptation for HTR), which builds upon the earlier DRANet framework. The process follows a four-stage pipeline:

  1. Source Pretraining: An HTR model (a CRNN) is trained on labeled source data.
  2. Visual Domain Adaptation: A generative network learns to separate images into content (the letters) and style (the visual appearance). It then recombines source content with target style to create "synthetic" training samples that look like the target domain but retain the source labels.
  3. Fine-Tuning: The HTR model is fine-tuned on these style-transferred images.
  4. Language Adaptation (DRAL-HTR): The authors introduce a post-processing step using the Qwen LLM to refine the HTR output, using neighboring lines and multiple recognition hypotheses as context.
Overview of the proposed domain-adaptive HTR pipeline.

Key Architectural Innovations:

  • ConvNeXt-V2 Encoder: Replaces simpler backbones to better capture multiscale features of complex handwriting.
  • FiLM (Feature-wise Linear Modulation): Used in the generator to precisely inject target style into the source content features at multiple stages.
  • PatchGAN Discriminator: Instead of judging an entire line as "real" or "fake," this evaluates local patches, which is better for capturing granular details like stroke width and ink texture.
  • Progressive Training: A strategy that delays the introduction of recognition-based losses until the generator has learned a basic style transfer, preventing noisy gradients from destabilizing the model.
Fig. 2: Overview of the proposed DRA-HTR architecture. The overall content–style adaptation princi- ple follows [1], while the architectures of its main components are redesigned in this work for HTR. The encoder (E) extracts visual features from source and target images, which are decomposed by the separa- tor (S) into content and style representations. The generator (G) recombines source content with target style to generate target-style handwritten text lines while preserving their textual content. The generated images are constrained using adversarial, reconstruction, latent-consistency, and perceptual objectives. DA is performed only once during training to adapt the HTR model to the target domain. After adap- tation, the model retains its original inference time.

Datasets

The framework was evaluated across seven HTR datasets spanning four languages:

  • English: IAM (modern) and Bentham (historical).
  • German: READ 2016 (historical).
  • Norwegian: NorHand (two distinct partitions).
  • Swedish: Riksarkivet (two distinct partitions).

These datasets provide a rigorous test of the model's ability to handle variations in language, historical period, and document degradation.

Key Insights and Ideas

  • Disentanglement is Key: By explicitly separating style from content, the model can generate training data that is visually indistinguishable from the target domain while maintaining 100% accurate labels.
  • Local Realism Matters: The switch to a PatchGAN discriminator highlights that for HTR, "realism" is found in the local textures of ink and paper rather than the global structure of the page.
  • Training Stability: One of the biggest contributions is the massive reduction in training variance. Previous generative DA methods were often "hit or miss"; DRA-HTR produces consistent results across independent runs.
  • LLMs as Contextual Correctors: Using an LLM to choose between the top 20 "hypotheses" from the HTR model (beam search) proved more effective than simple greedy decoding, especially when the LLM was allowed to see the preceding and following lines.

Key Results

  • Superior Accuracy: DRA-HTR reduced the average Character Error Rate (CER) by 13.52 percentage points compared to the previous state-of-the-art domain adaptation method.
  • Robustness: The average standard deviation across training runs decreased from 7.23 to 1.11 percentage points, indicating a much more stable training process.
  • LLM Boost: The addition of LLM-based post-correction (DRAL-HTR) further reduced the CER by an average of 2.16 percentage points.
  • Zero-Shot vs. Adapted: In one example (IAM to Bentham), a standard model had a 13.23% CER; after DRA-HTR adaptation, this dropped to 5.29%, and further to 3.63% with LLM correction.

Limitations and Future Work

  • Computational Cost: Using a large LLM like Qwen for post-correction is computationally expensive and significantly slower than the HTR model's native inference.
  • Language Bias: The LLM improvements were most pronounced in English. The authors noted that gains were more limited for other languages, likely due to the LLM's training distribution or the specific historical dialects involved.
  • Future Directions: The authors suggest exploring more efficient language adaptation strategies that don't rely on massive LLMs and further optimizing hyperparameters for specific language-target pairs.

Summary updated

Abstract

Handwritten text recognition (HTR), particularly when applied to historical documents, remains challenging due to the scarcity of annotated data and the substantial domain shifts caused by variations in handwriting styles, document degradation, acquisition conditions, and linguistic characteristics. While large amounts of unlabeled target-domain data are often available, effectively exploiting them requires adaptation strategies tailored to the specific requirements of HTR. We propose DRA-HTR, an unsupervised generative domain adaptation pipeline specifically designed for HTR and built upon the DRANet framework. The proposed approach combines source-domain pretraining, unsupervised adaptation to the visual characteristics of the target domain, relying on the generation of target-style source samples for task-specific training. We further investigate the integration of post-HTR correction to complement visual adaptation with language adaptation, relying on large language models, which we introduce as DRAL-HTR. The proposed framework is evaluated across seven HTR datasets spanning four languages, using a single set of hyperparameters for all source-target combinations. The recognition model achieves state-of-the-art performance on four datasets and competitive results on the remaining three. In addition, the proposed DRA-HTR establishes a new state-of-the-art for domain adaptation across all seven datasets and substantially improves overall robustness and training stability compared to the original DRANet framework: it reduces the Character Error Rate (CER) in average by 13.52 percentage points, and the average standard deviation decreases from 7.23 to 1.11 percentage points. Finally, post-HTR correction further reduces the CER by 2.16 percentage points.
BibTeX
@misc{pathirage2026dra,
  author = {Gayan H. Pathirage and Florent Imbert and Denis Coquenet and Elisa H. Barney Smith},
  title = {{DRA-HTR: Disentangling Representation and Adaptation for Handwritten Text Recognition}},
  year = {2026},
  eprint = {2611.18143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18143},
}
Figure