Domain Adaptation Based Pipeline for Character Classification and Handwritten Text Recognition

Florent Imbert, Simon Corbillé, Hui Han, Elisa H. Barney SmitharXiv 2026 · DOI · arXiv

TL;DR

This paper introduces a novel unsupervised domain adaptation (UDA) pipeline that bridges the gap between labeled source data and unlabeled target data for character and text recognition. By disentangling "content" from "style" and retraining an expert model on source data transformed into the target's visual style, the authors achieved a new state-of-the-art on the Safran-MNIST-DLS dataset and competitive results on historical manuscripts without using a single target label.

What Problem Does This Paper Address?

Deep learning models for Handwritten Text Recognition (HTR) and character classification are notoriously data-hungry. While we have massive labeled datasets for standard tasks (like EMNIST for letters), these models often fail when applied to specialized domains—such as serial numbers engraved on aircraft parts or 18th-century German manuscripts—due to "domain shift."

The visual properties (style) of the target domain differ wildly from the training data (source), yet the underlying characters (content) remain the same. Manually labeling these specialized datasets is expensive and requires expert knowledge. The authors tackle the challenge of leveraging the vast amounts of unlabeled target data available in industrial and academic settings to improve model generalization without additional manual effort.

Methods

The authors propose a three-step pipeline designed to align feature distributions between domains:

  1. Supervised Pretraining: An "Expert Model" is trained on labeled source data to learn a robust general representation. For character classification, they use a ResNet-34; for line-level HTR, they use a Convolutional Recurrent Neural Network (CRNN).
  2. Unsupervised Domain Adaptation: They employ a Disentangling Representation and Adaptation Network (DRANet). This architecture uses an encoder-separator-generator structure to isolate the "content" (the structure of the character) from the "style" (the texture, lighting, and noise of the image).
  3. Retraining on Adapted Data: The DRANet generates a new dataset: the original source content rendered in the target domain's style. The Expert Model is then retrained on this "stylized" source data, effectively teaching it to recognize familiar characters in an unfamiliar visual context.
Overview of the three-step pipeline: pretraining, domain adaptation via style transfer, and final retraining.

Crucially, the DRANet is made task-specific by using the initial layers of the Expert Model as its encoder, ensuring that the features being disentangled are relevant to the final recognition task.

The DRANet architecture disentangles content and style to generate cross-domain images for training.

Datasets

The pipeline was evaluated on two distinct tasks:

  • Character Classification: The target was the Safran-MNIST-DLS dataset (engraved dot-style serial numbers). Source data included EMNIST and a custom Synthetic dataset designed to mimic engraved textures.
  • Handwritten Line Recognition: The authors used the ICFHR 2018 READ dataset. The source was a generic collection of 17 documents, while the targets were five specific historical documents (e.g., Konzilprotokolle C, Schiller) with unique handwriting styles and languages (German and Italian).

Key Insights and Ideas

  • Disentanglement is the "Aha!" Moment: Instead of just trying to make features "look" the same in a latent space, the model explicitly separates what is written from how it is written. This allows the generator to create high-quality training samples that maintain semantic integrity while adopting target-domain aesthetics.
  • Stylized Retraining vs. Feature Alignment: Many UDA methods focus on making the internal feature maps of the source and target indistinguishable. This paper shows that explicitly retraining the classifier on "translated" images is a powerful way to bridge the domain gap.
  • Zero-Label Superiority: In the HTR experiments, the UDA pipeline (using 0 target labels) often outperformed supervised models trained on 16 pages of manually labeled target data. This suggests that the diversity of the "stylized" source data can be more valuable than a small amount of real, labeled target data.
  • No Inference Penalty: While the training process is more complex due to the DRANet, the final inference is performed solely by the Expert Model. This means the performance gains come with zero cost to processing speed in production.

Key Results

  • State-of-the-Art (SOTA): On the Safran-MNIST-DLS classification task, the pipeline achieved a 76.17% Macro F1-score, outperforming the previous best competition entry by 11 percentage points.
  • HTR Performance: On the READ 2018 benchmark, the method significantly reduced the Character Error Rate (CER). For the Schiller document, the CER dropped from 16.75% (no adaptation) to 1.89% (with UDA), beating the supervised SOTA of 7.64%.
  • Efficiency: The final model can process ~13,000 character images per second or ~1,700 text lines per second on an NVIDIA A100 GPU.

Limitations and Future Work

The authors noted that the pipeline is sensitive to linguistic shifts. Performance was lower on the Ricordi dataset (Italian) when the source data was primarily German/English. This indicates that while the model adapts well to visual style, it still struggles when the underlying "content" distribution (character frequencies and word structures) changes significantly.

Future work may involve extending this pipeline to page-level recognition and exploring ways to adapt to new languages or vocabularies within the same unsupervised framework.

Summary updated

Abstract

Character classification and handwritten text recognition, particularly in specialized tasks, often suffer from limited labeled data, while a large amount of unlabeled data remains unused. To address this challenge, we propose a novel unsupervised domain adaptation-based pipeline to reduce the distribution gap between source and target domains. Our pipeline follows three main steps. In the first step, we pretrained an expert model on source data, to learn a strong general representation. In the second step, we perform unsupervised domain adaptation to leverage unlabeled target data, adapting the model to the specific characteristics of the target domain. In the third step, we trained the expert model on source data in the target style. By aligning the feature distributions between the source and target, our method enables the model to generalize effectively across diverse datasets. The unsupervised training increases the computation time only during training not in inference. We evaluate the pipeline on both character-level and historical handwritten line recognition tasks, demonstrating its flexibility. We establish a new state-of-the-art on the Safran-MNIST-DLS dataset defined in Track 2 of the DAGECC competition with 76.17% Macro F1-score and achieve competitive results on each of the specific datasets of READ 2018 without using the labels of the target data. Code is available at https://github.com/simon-corbi/DRANet-for-classification-and-htr
BibTeX
@misc{imbert2026domain,
  author = {Florent Imbert and Simon Corbillé and Hui Han and Elisa H. Barney Smith},
  title = {{Domain Adaptation Based Pipeline for Character Classification and Handwritten Text Recognition}},
  year = {2026},
  eprint = {2611.17952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17952},
}
Figure