Towards the Unseen: Iterative Text Recognition by Distilling from Errors

Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Aneeshan Sain, Yi-Zhe SongICCV 2021 · OpenReview

Introduction

State-of-the-art Scene Text Recognition (STR) models often struggle with "unseen" words—sequences of characters not present in the training vocabulary. While these models perform well on common words, their accuracy drops significantly when encountering rare terms, technical jargon, or brand names. This "vocabulary-locked" behavior stems from the single-pass nature of current architectures, which lack a mechanism for self-correction. In contrast, humans recognize unfamiliar words through iterative refinement. This paper proposes a meta-framework that introduces an iterative feedback loop to improve generalization to unseen character sequences.

Methodology

The proposed framework treats text recognition as a multi-step refinement process. The architecture is divided into three primary components:

  1. Main Network ($T$): Comprising a feature extractor ($T_A$) and a decoder ($T_B$).
  2. Feedback Module ($F$): A Cross-Modal Conditional Variational Autoencoder (CVAE) that serves as the core of the refinement process. It maps discrete text predictions from a previous step back into a continuous parameter space.
  3. Feedback Conditioning Layer: This layer uses parameters from the CVAE to modulate visual feature maps (via FiLM-like modulation) before they are re-processed by the decoder in the subsequent iteration.
Architecture of the proposed iterative framework during training and inference.

A key innovation is the "Distilling from Errors" strategy. The feedback module is trained not only on ground-truth data but also on realistic error distributions ($\mathcal{G}$) collected from other high-performing models. By learning from these common OCR mistakes (e.g., confusing 'h' with 'n'), the CVAE develops a linguistic prior that guides the model from an incorrect prediction toward the correct one.

Experimental Setup

The model was evaluated on standard Scene Text Recognition (STR) datasets (IIIT5K, SVT, IC13, IC15, SVTP, CUTE80) and Handwriting Text Recognition (HTR) datasets (IAM, RIMES). To rigorously test generalization, the authors introduced a Disjoint Setup (DS), where the test set contains only words entirely absent from the training set. This setup provides a more realistic measure of a model's ability to "read" rather than memorize.

Results

The iterative framework demonstrates consistent performance improvements, particularly in the challenging Disjoint Setup.

Performance comparison between baseline and proposed models on standard and disjoint setups across STR and HTR datasets.

Key findings include:

  • Significant Gains in DS: On the IAM handwriting dataset, the framework achieved a 10.5% absolute increase in Word Recognition Accuracy (WRA) over the baseline. Similar gains were observed on RIMES (10.8%) and IC15 (6.9%).
  • Iterative Refinement: Accuracy typically peaks at the second iteration ($t=2$), showing the effectiveness of the feedback loop.
  • Efficiency: The feedback module is lightweight, adding only approximately 2.19ms of computational overhead per iteration.
  • Versatility: The module acts as a "plug-and-play" meta-framework, successfully boosting the performance of various existing SOTA architectures like ASTER and SCATTER.

Discussion and Conclusion

While the iterative approach significantly improves recognition of unseen words, it faces diminishing returns beyond a few iterations and introduces some computational overhead compared to single-pass models. Furthermore, the feedback quality is tied to the representativeness of the error distribution $\mathcal{G}$. Despite these limitations, the framework represents a significant step toward more robust, human-like text recognition systems that can generalize beyond their training vocabulary.

Summary updated

Abstract

Visual text recognition is undoubtedly one of the most extensively researched topics in computer vision. Great progress have been made to date, with the latest models starting to focus on the more practical "in-the-wild" setting. However, a salient problem still hinders practical deployment -- prior arts mostly struggle with recognising unseen (or rarely seen) character sequences. In this paper, we put forward a novel framework to specifically tackle this "unseen" problem. Our framework is iterative in nature, in that it utilises predicted knowledge of character sequences from a previous iteration, to augment the main network in improving the next prediction. Key to our success is a unique cross-modal variational autoencoder to act as a feedback module, which is trained with the presence of textual error distribution data. This module importantly translate a discrete predicted character space, to a continuous affine transformation parameter space used to condition the visual feature map at next iteration. Experiments on common datasets have shown competitive performance over state-of-the-arts under the conventional setting. Most importantly, under the new disjoint setup where train-test labels are mutually exclusive, ours offers the best performance thus showcasing the capability of generalising onto unseen words.
BibTeX
@inproceedings{bhunia2021towards,
  author = {Ayan Kumar Bhunia and Pinaki Nath Chowdhury and Aneeshan Sain and Yi-Zhe Song},
  title = {{Towards the Unseen: Iterative Text Recognition by Distilling from Errors}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=RBAaqnK-Zt0},
}
Figure