SCAN: Sliding Convolutional Attention Network for Scene Text Recognition

Yi-Chao Wu, Fei Yin, Xu-Yao Zhang, Li Liu, Cheng-Lin LiuarXiv 2018 · OpenReview

TL;DR

This paper introduces SCAN, a novel scene text recognition system that mimics human reading by using a sliding window and attention mechanism. SCAN offers improved interpretability and significantly faster training times compared to traditional recurrent neural network (RNN) based methods by leveraging a fully convolutional architecture.

What Problem Does This Paper Address?

Recognizing text in natural images (scene text recognition) is a challenging but crucial task for applications like image retrieval and autonomous navigation. Existing state-of-the-art methods often rely on Recurrent Neural Networks (RNNs) which, while effective, suffer from a lack of transparency ("black box" nature) and slow training due to sequential processing. This makes it difficult to understand why a model makes a certain prediction and hinders further improvement. Furthermore, RNNs struggle with parallel computation, making training time-consuming.

Methods

The SCAN (Sliding Convolutional Attention Network) system is designed to address these limitations by drawing inspiration from human reading. It consists of three main components:

  • Sliding Window Layer: This layer mimics the "saccades" and "fixations" of human eye movements during reading. It splits the textline into overlapping windows of varying scales, allowing the model to capture context at different levels, similar to how our eyes focus on specific areas while scanning.
  • Convolutional Feature Extractor: A deep convolutional neural network (CNN) processes these windows to extract discriminative features. This is a standard approach for image feature extraction.
  • Convolutional Sequence to Sequence Learning Module: This module, inspired by recent advancements in machine translation, uses a fully convolutional encoder-decoder architecture. Crucially, it replaces the sequential processing of RNNs with convolutional operations, enabling parallel computation during training. An attention mechanism is incorporated to allow the decoder to focus on relevant parts of the extracted features, mimicking human attention during reading.

The key novelty lies in the combination of a biologically inspired sliding window mechanism with a fully convolutional sequence-to-sequence model, which allows for parallel training and improved interpretability.

Datasets

The paper uses a synthetic dataset of 8 million images for training. For evaluation, it tests SCAN on four standard scene text recognition benchmarks:

  • IIIT 5k-word (IIIT5k)
  • Street View Text (SVT)
  • ICDAR 2003 (IC03)
  • ICDAR 2013 (IC13)

Key Insights and Ideas

  • Human Reading as Inspiration: The core idea is that mimicking human reading mechanisms (saccades, fixations, and attention) can lead to more interpretable and effective text recognition models.
  • Parallelism for Speed: By adopting a fully convolutional approach for sequence learning, SCAN achieves significant speedups in training compared to RNN-based methods, as computations can be parallelized across the sequence.
  • Interpretability through Attention: The attention mechanism in SCAN provides visual insights into which parts of the input image the model is focusing on when making predictions, offering a degree of transparency lacking in many RNN models.
  • Multi-Scale Windows for Robustness: Using sliding windows of multiple scales helps the model adapt to varying character fonts and sizes, improving its ability to handle diverse scene text.

Key Results

SCAN demonstrates state-of-the-art performance on several challenging benchmarks.

  • Performance: On the IIIT5k dataset, SCAN achieved 99.1% accuracy with multi-scale windows, outperforming previous methods. It also showed strong results on SVT, IC03, and IC13.
  • Speed: The paper claims SCAN is at least 9 times faster than recurrent models in training speed due to its parallelizable architecture. It can recognize an image in approximately 0.3 seconds.
  • Interpretability: Visualizations of attention weights show clear focus on relevant image regions, providing insights into the model's decision-making process.

Limitations and Future Work

  • Handling Irregular Text: While SCAN performs well, it still struggles with severely blurred or curved word images, and attention can be "drifted" in cases of insertion or deletion errors.
  • Future Directions: The authors suggest exploring more advanced feature extraction techniques (like DenseNet) to enhance feature discriminability for irregular text. They also propose using curve fitting techniques to better handle text lines with complex shapes. Further work could also focus on improving the model's performance on more challenging datasets.

Summary updated

Abstract

Scene text recognition has drawn great attentions in the community of computer vision and artificial intelligence due to its challenges and wide applications. State-of-the-art recurrent neural networks (RNN) based models map an input sequence to a variable length output sequence, but are usually applied in a black box manner and lack of transparency for further improvement, and the maintaining of the entire past hidden states prevents parallel computation in a sequence. In this paper, we investigate the intrinsic characteristics of text recognition, and inspired by human cognition mechanisms in reading texts, we propose a scene text recognition method with sliding convolutional attention network (SCAN). Similar to the eye movement during reading, the process of SCAN can be viewed as an alternation between saccades and visual fixations. Compared to the previous recurrent models, computations over all elements of SCAN can be fully parallelized during training. Experimental results on several challenging benchmarks, including the IIIT5k, SVT and ICDAR 2003/2013 datasets, demonstrate the superiority of SCAN over state-of-the-art methods in terms of both the model interpretability and performance.
BibTeX
@misc{wu2018scan,
  author = {Yi-Chao Wu and Fei Yin and Xu-Yao Zhang and Li Liu and Cheng-Lin Liu},
  title = {{SCAN: Sliding Convolutional Attention Network for Scene Text Recognition}},
  year = {2018},
  eprint = {1806.00578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.00578},
}
Figure