Attacking Optical Character Recognition (OCR) Systems with Adversarial Watermarks

Lu Chen, Wei Xu

Abstract

Optical character recognition (OCR) is widely applied in real applications serving as a key preprocessing tool. The adoption of deep neural network (DNN) in OCR results in the vulnerability against adversarial examples which are crafted to mislead the output of the threat model. Different from vanilla colorful images, images of printed text have clear backgrounds usually. However, adversarial examples generated by most of the existing adversarial attacks are unnatural and pollute the background severely. To address this issue, we propose a watermark attack method to produce natural distortion that is in the disguise of watermarks and evade human eyes' detection. Experimental results show that watermark attacks can yield a set of natural adversarial examples attached with watermarks and attain similar attack performance to the state-of-the-art methods in different attack scenarios.

Introduction

Most adversarial attacks—techniques used to trick AI by adding subtle noise to an image—were originally designed for natural photos like animals or landscapes. When applied to Optical Character Recognition (OCR), these methods often fail a crucial "sanity check": human perception. OCR typically processes text on clean, white backgrounds, where standard adversarial noise creates a "dirty" or "smudged" look that is immediately suspicious to a human reader.

Furthermore, modern OCR systems are not simple classifiers; they utilize end-to-end neural networks, often paired with Connectionist Temporal Classification (CTC), to read entire sequences of characters. This makes them harder to attack than single-object images because perturbations must span multiple characters and account for surrounding context to successfully mislead the model.

Methodology: The WATERMARK Attack

To address the limitations of traditional noise, we introduce the WATERMARK attack. This method builds upon the Momentum Iterative Method (MIM) but introduces a spatial constraint: instead of spreading noise across the entire image, perturbations are restricted to a predefined region $\Omega$—the shape of a natural-looking watermark or stamp.

Overview of the WATERMARK attack pipeline.

We developed four distinct variants to maximize visual naturalness:

  1. WM (Standard): Restricts noise strictly to the watermark area.
  2. WMinit: Starts with a visible watermark already "pasted" onto the image to ensure the final result looks like a complete, solid mark.
  3. WMneg: Only allows the attack to make pixels darker. Since making a white background "whiter" is impossible, only using negative gradients creates a more realistic "ink" effect.
  4. WMedge: Confines the attack to the edges of the characters themselves, mimicking common printing or scanning defects.

Experimental Evaluation

The methods were evaluated using a massive dataset of 3.64 million Chinese text images. For attack simulations, 158 sentences were selected from the People’s Daily corpus, focusing on 691 pairs of visually similar "antonym characters" (e.g., changing one character to its opposite) to ensure the adversarial text remained semantically plausible.

Visual examples of adversarial text substitutions and their corresponding OCR outputs.

Key Results

  • Effectiveness: While the standard MIM attack achieved a 92.4% targeted success rate, it significantly degraded image quality. In contrast, the WMedge variant achieved a high 87.3% success rate while maintaining superior visual fidelity.
  • Transferability: The attacks demonstrated "black-box" capability. Adversarial images generated to fool a DenseNet model also successfully misled the widely-used Tesseract OCR in many cases.
  • Real-World Impact: The study demonstrated successful attacks on sensitive documents, including a New Hampshire driver's license (altering the ID number) and corporate annual reports (modifying financial revenue figures).
  • Defense Resistance: The attacks proved robust against common defenses like JPEG compression and blurring. While "inpainting" (digitally removing the watermark) could mitigate the attack, it often accidentally deleted parts of the actual text, rendering the document unreadable.

Discussion and Conclusion

The WATERMARK attack exploits a human psychological blind spot: we are conditioned to ignore watermarks and logos on official documents. By hiding malicious perturbations within these familiar elements, the attack achieves high success rates without alerting human observers.

Future work will explore dynamic placement for watermarks and the integration of Language Models to ensure substituted text is grammatically perfect. These findings highlight a critical need for new defense mechanisms, as traditional image-processing defenses are currently insufficient to protect OCR systems from these sophisticated semantic attacks.

Summary updated

Abstract

Optical character recognition (OCR) is widely applied in real applications serving as a key preprocessing tool. The adoption of deep neural network (DNN) in OCR results in the vulnerability against adversarial examples which are crafted to mislead the output of the threat model. Different from vanilla colorful images, images of printed text have clear backgrounds usually. However, adversarial examples generated by most of the existing adversarial attacks are unnatural and pollute the background severely. To address this issue, we propose a watermark attack method to produce natural distortion that is in the disguise of watermarks and evade human eyes' detection. Experimental results show that watermark attacks can yield a set of natural adversarial examples attached with watermarks and attain similar attack performance to the state-of-the-art methods in different attack scenarios.
BibTeX
@misc{chen2020attacking,
  author = {Lu Chen and Wei Xu},
  title = {{Attacking Optical Character Recognition (OCR) Systems with Adversarial Watermarks}},
  year = {2020},
  eprint = {2002.03095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.03095},
}
Figure