TL;DR
This paper introduces a novel unsupervised domain adaptation (UDA) pipeline that bridges the gap between labeled source data and unlabeled target data for character and text recognition. By disentangling "content" from "style" and retraining an expert model on source data transformed into the target's visual style, the authors achieved a new state-of-the-art on the Safran-MNIST-DLS dataset and competitive results on historical manuscripts without using a single target label.
What Problem Does This Paper Address?
Deep learning models for Handwritten Text Recognition (HTR) and character classification are notoriously data-hungry. While we have massive labeled datasets for standard tasks (like EMNIST for letters), these models often fail when applied to specialized domains—such as serial numbers engraved on aircraft parts or 18th-century German manuscripts—due to "domain shift."
The visual properties (style) of the target domain differ wildly from the training data (source), yet the underlying characters (content) remain the same. Manually labeling these specialized datasets is expensive and requires expert knowledge. The authors tackle the challenge of leveraging the vast amounts of unlabeled target data available in industrial and academic settings to improve model generalization without additional manual effort.
Methods
The authors propose a three-step pipeline designed to align feature distributions between domains:
- Supervised Pretraining: An "Expert Model" is trained on labeled source data to learn a robust general representation. For character classification, they use a ResNet-34; for line-level HTR, they use a Convolutional Recurrent Neural Network (CRNN).
- Unsupervised Domain Adaptation: They employ a Disentangling Representation and Adaptation Network (DRANet). This architecture uses an encoder-separator-generator structure to isolate the "content" (the structure of the character) from the "style" (the texture, lighting, and noise of the image).
- Retraining on Adapted Data: The DRANet generates a new dataset: the original source content rendered in the target domain's style. The Expert Model is then retrained on this "stylized" source data, effectively teaching it to recognize familiar characters in an unfamiliar visual context.
Crucially, the DRANet is made task-specific by using the initial layers of the Expert Model as its encoder, ensuring that the features being disentangled are relevant to the final recognition task.
Datasets
The pipeline was evaluated on two distinct tasks:
- Character Classification: The target was the Safran-MNIST-DLS dataset (engraved dot-style serial numbers). Source data included EMNIST and a custom Synthetic dataset designed to mimic engraved textures.
- Handwritten Line Recognition: The authors used the ICFHR 2018 READ dataset. The source was a generic collection of 17 documents, while the targets were five specific historical documents (e.g., Konzilprotokolle C, Schiller) with unique handwriting styles and languages (German and Italian).
Key Insights and Ideas
- Disentanglement is the "Aha!" Moment: Instead of just trying to make features "look" the same in a latent space, the model explicitly separates what is written from how it is written. This allows the generator to create high-quality training samples that maintain semantic integrity while adopting target-domain aesthetics.
- Stylized Retraining vs. Feature Alignment: Many UDA methods focus on making the internal feature maps of the source and target indistinguishable. This paper shows that explicitly retraining the classifier on "translated" images is a powerful way to bridge the domain gap.
- Zero-Label Superiority: In the HTR experiments, the UDA pipeline (using 0 target labels) often outperformed supervised models trained on 16 pages of manually labeled target data. This suggests that the diversity of the "stylized" source data can be more valuable than a small amount of real, labeled target data.
- No Inference Penalty: While the training process is more complex due to the DRANet, the final inference is performed solely by the Expert Model. This means the performance gains come with zero cost to processing speed in production.
Key Results
- State-of-the-Art (SOTA): On the Safran-MNIST-DLS classification task, the pipeline achieved a 76.17% Macro F1-score, outperforming the previous best competition entry by 11 percentage points.
- HTR Performance: On the READ 2018 benchmark, the method significantly reduced the Character Error Rate (CER). For the Schiller document, the CER dropped from 16.75% (no adaptation) to 1.89% (with UDA), beating the supervised SOTA of 7.64%.
- Efficiency: The final model can process ~13,000 character images per second or ~1,700 text lines per second on an NVIDIA A100 GPU.
Limitations and Future Work
The authors noted that the pipeline is sensitive to linguistic shifts. Performance was lower on the Ricordi dataset (Italian) when the source data was primarily German/English. This indicates that while the model adapts well to visual style, it still struggles when the underlying "content" distribution (character frequencies and word structures) changes significantly.
Future work may involve extending this pipeline to page-level recognition and exploring ways to adapt to new languages or vocabularies within the same unsupervised framework.
Summary updated