Sequence-To-Sequence Domain Adaptation Network for Robust Text Image Recognition

Yaping Zhang, Shuai Nie, Wenju Liu, Xing Xu, Dongxiang Zhang, Heng Tao Shen

Abstract

Domain adaptation has shown promising advances for alleviating domain shift problem. However, recent visual domain adaptation works usually focus on non-sequential object recognition with a global coarse alignment, which is inadequate to transfer effective knowledge for sequence-like text images with variable-length fine-grained character information. In this paper, we develop a Sequence-to-Sequence Domain Adaptation Network (SSDAN) for robust text image recognition, which could exploit unsupervised sequence data by an attention-based sequence encoder-decoder network. In the SSDAN, a gated attention similarity (GAS) unit is introduced to adaptively focus on aligning the distribution of the source and target sequence data in an attended character-level feature space rather than a global coarse alignment. Extensive text recognition experiments show the SSDAN could efficiently transfer sequence knowledge and validate the promising power of the proposed model towards real world applications in various recognition scenarios, including the natural scene text, handwritten text and even mathematical expression recognition.

Introduction

Standard Optical Character Recognition (OCR) models often struggle with "domain shift"—the performance drop that occurs when a model trained on one type of data (e.g., clean synthetic fonts) is tested on another (e.g., blurry street signs or messy handwriting). While Unsupervised Domain Adaptation (UDA) exists to solve this, most methods focus on global image alignment. This works for simple object classification but fails for text, which is sequential and variable-length. A global "average" of a word image loses the fine-grained details of individual characters.

Examples of domain shift across scene text, handwritten text, and mathematical expressions.

The Proposed SSDAN Framework

The authors propose the Sequence-to-Sequence Domain Adaptation Network (SSDAN), which consists of a standard encoder-decoder backbone enhanced by a specialized adaptation unit. The architecture uses a CNN (DenseNet) to extract features, a Bi-LSTM to capture context, and an attention-based GRU decoder to predict character sequences.

The SSDAN architecture featuring the Gated Attention Similarity (GAS) unit for character-level alignment.

The core innovation is the Gated Attention Similarity (GAS) Unit. Instead of aligning the entire image, the GAS unit uses the model's own attention mechanism to "pick out" character-specific features. Not all attention steps are useful; if the model is looking at background noise or a space between letters, trying to align those features across domains can introduce errors. The GAS unit uses a confidence threshold ($p_c$) to only perform domain alignment if the model is reasonably sure it is currently "attending" to a valid character. To minimize the distance between the source and target distributions, the authors use Correlation Alignment (CORAL), which aligns the second-order statistics (covariances) of the attended features.

Experimental Evaluation

The researchers evaluated SSDAN across three distinct text recognition scenarios:

  • Scene Text: Source data was MJSYNTH (8 million synthetic images). Target datasets included IIIT5K, Street View Text (SVT), ICDAR-2003, and ICDAR-2013.
  • Handwritten Text: Evaluated on the IAM dataset, focusing on writer-independent recognition.
  • Mathematical Expressions: Evaluated on CROHME 2014, involving complex structural recognition of LaTeX-style notations.

Key Results and Insights

SSDAN consistently outperformed the non-adapted baseline. On the IIIT5K scene text dataset, accuracy improved from 81.1% to 83.8%. On the IAM handwriting dataset, it achieved a Word Error Rate (WER) of 22.2%, outperforming several specialized handwriting models. For mathematical expressions, the model achieved an Expression Recognition Rate (ExpRate) of 41.6% on CROHME 2014.

The most clever insight is using the attention mechanism as a dynamic "cropper." By aligning only the features the attention head is currently focused on, the model effectively performs character-level adaptation without needing character-level bounding boxes. Furthermore, the "Gated" part of GAS is vital; blindly aligning all features (including noise) actually degrades performance. Filtering for high-confidence character features ensures the model learns domain-invariant content rather than noise.

Conclusion and Future Work

While SSDAN shows significant promise, it has limitations. The gate mechanism relies on the model's internal confidence; if the initial source model is very poor, the adaptation process might stall. Additionally, while the model handled math expressions well, mathematical structures are significantly more diverse than linear text, suggesting future work could better handle non-linear, 2D structures.

Summary updated

Abstract

Domain adaptation has shown promising advances for alleviating domain shift problem. However, recent visual domain adaptation works usually focus on non-sequential object recognition with a global coarse alignment, which is inadequate to transfer effective knowledge for sequence-like text images with variable-length fine-grained character information. In this paper, we develop a Sequence-to-Sequence Domain Adaptation Network (SSDAN) for robust text image recognition, which could exploit unsupervised sequence data by an attention-based sequence encoder-decoder network. In the SSDAN, a gated attention similarity (GAS) unit is introduced to adaptively focus on aligning the distribution of the source and target sequence data in an attended character-level feature space rather than a global coarse alignment. Extensive text recognition experiments show the SSDAN could efficiently transfer sequence knowledge and validate the promising power of the proposed model towards real world applications in various recognition scenarios, including the natural scene text, handwritten text and even mathematical expression recognition.
BibTeX
@inproceedings{zhang2019sequence,
  author = {Yaping Zhang and Shuai Nie and Wenju Liu and Xing Xu and Dongxiang Zhang and Heng Tao Shen},
  title = {{Sequence-To-Sequence Domain Adaptation Network for Robust Text Image Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=kPzxhc6U2p},
}
Figure