Stroke-based Character Recognition with Deep Reinforcement Learning

Zhewei Huang, Wen Heng, Yuanzheng Tao, Shuchang Zhou

Abstract

Background elimination for noisy character images or character images from real scene is still a challenging problem, due to the bewildering backgrounds, uneven illumination, low resolution and different distortions. We propose a stroke-based character reconstruction (SCR) method that use a weighted quadratic Bezier curve (WQBC) to represent strokes of a character. Only training on our synthetic data, our stroke extractor can achieve excellent reconstruction effect in real scenes. Meanwhile, it can also help achieve great ability in defending adversarial attacks of character recognizers.

Introduction

Character recognition in natural environments, such as street numbers or signage, remains a significant challenge for computer vision. Traditional Deep Neural Networks (DNNs) often struggle with uneven lighting, low resolution, and complex backgrounds. Furthermore, these models are notoriously vulnerable to adversarial attacks—subtle pixel-level perturbations that can lead to catastrophic misclassification despite being invisible to the human eye.

The core issue is that standard models treat images as collections of independent pixels rather than structured sequences of strokes. By shifting the focus to structural information, we can develop systems that are inherently more robust to noise and intentional interference.

Methodology: Stroke-based Character Reconstruction (SCR)

The proposed SCR method utilizes an encoder-decoder architecture designed with a structural bottleneck. Instead of reconstructing pixels directly, the model represents characters as a set of Weighted Quadratic Bezier Curves (WQBC).

Stroke Representation

Each stroke is defined by nine parameters: three $(x, y)$ coordinate points and their associated thicknesses ($w$). This compact representation forces the model to capture the essential geometry of the character.

The Pipeline

  1. Stroke Extractor (Encoder): A VGG-like neural network processes the input image to predict parameters for a fixed number of strokes (e.g., four curves per character).
  2. Differentiable Decoder: A pre-trained, differentiable decoder converts these parameters back into a clean image. This allows the entire pipeline to be trained via backpropagation using an $L_2$ loss between the reconstructed image and a clean ground truth.
Figure 4: Stroke extraction and binarization process on real-world SVHN images.

By forcing the output through this "stroke bottleneck," the system acts as a structural low-pass filter. Since the decoder can only draw smooth curves, it naturally ignores pixel-level adversarial noise.

Experimental Setup

The model was evaluated on several datasets:

  • MNIST: Used primarily to test resilience against adversarial attacks.
  • SVHN (Street View House Numbers): A challenging real-world dataset of house numbers.
  • Synthetic Data: To overcome the lack of labeled stroke data, the authors generated 200,000 synthetic images using 250 fonts and various augmentations (rotation, blur, and noise).

Key Results

Adversarial Defense

The SCR method demonstrates remarkable robustness. On the MNIST dataset, it achieved an average accuracy of 88.60% across various adversarial attacks, including FGSM, BIM, and JSMA. In contrast, undefended models dropped to 13% accuracy. This performance is nearly on par with human performance (88.90%).

Figure 5: Comparison of original and adversarial images (top) with their SCR reconstructions (bottom).

Real-World Generalization

Despite being trained exclusively on synthetic data, the model achieved an 89% reconstruction success rate on the SVHN test set. This "zero-shot" generalization suggests that the "language of strokes" is a more universal and robust feature set than raw pixel patterns.

Conclusion and Future Work

The SCR method provides a powerful defense against adversarial attacks and background noise by leveraging structural priors. While highly effective for digits and simple characters, future work will explore scaling this approach to more complex scripts (like Chinese) and implementing dynamic stroke counts using Reinforcement Learning to handle varying character complexity.

Reference: Huang, Z., Heng, W., Tao, Y., & Zhou, S. (2018). Stroke-based Character Reconstruction. arXiv:1806.08990v3 [cs.CV].

Summary updated

Abstract

Background elimination for noisy character images or character images from real scene is still a challenging problem, due to the bewildering backgrounds, uneven illumination, low resolution and different distortions. We propose a stroke-based character reconstruction(SCR) method that use a weighted quadratic Bezier curve(WQBC) to represent strokes of a character. Only training on our synthetic data, our stroke extractor can achieve excellent reconstruction effect in real scenes. Meanwhile. It can also help achieve great ability in defending adversarial attacks of character recognizers.
BibTeX
@misc{huang2018stroke,
  author = {Zhewei Huang and Wen Heng and Yuanzheng Tao and Shuchang Zhou},
  title = {{Stroke-based Character Recognition with Deep Reinforcement Learning}},
  year = {2018},
  eprint = {1806.08990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.08990},
}
Figure