DeepErase: Weakly Supervised Ink Artifact Removal in Document Text Images

Yike Qi, W. Ronny Huang, Qianqian Li, Jonathan DegangeWACV 2020

TL;DR

This paper introduces DeepErase, a novel deep learning approach to automatically remove distracting ink artifacts (like underlines, boxes, and smudges) from document images. By using a weakly supervised method to generate training data, DeepErase significantly improves the accuracy of optical character recognition (OCR) systems on "dirty" documents, making them more robust to real-world imperfections.

What Problem Does This Paper Address?

Despite decades of digitization, many industries still rely on paper documents, necessitating robust Optical Character Recognition (OCR) for data extraction. While OCR technology has advanced significantly, its performance is severely hampered by "ink artifacts" – elements like underlines, fill-in-the-blank boxes, random smudges, or stray marks that are common in real-world documents. These artifacts can overlap with text, making it difficult for OCR systems to accurately transcribe the content. Existing OCR datasets often feature clean documents, leading to models that perform poorly on the messy, real-world documents encountered in practice. While some solutions exist, they often require manual annotation or are not robust to the variety of artifacts found in documents.

Methods

The core of DeepErase is a fully convolutional neural network, specifically a U-Net architecture, designed for semantic segmentation. This network takes a document image as input and outputs a pixel-wise mask indicating the presence of ink artifacts. The key innovation lies in the weakly supervised training approach. Instead of requiring manual pixel-level annotations, the authors developed a programmatic method to assemble realistic-looking "dirty" text images. This involves taking clean text images and overlaying various types of artifacts (underlines, boxes, smudges, spurious strokes) at random offsets. The segmentation mask is automatically generated during this assembly process. Once the network is trained to predict these artifact masks, it can be used during inference to identify and remove the artifacts from new images, effectively "cleansing" them before they are fed into an OCR engine.

Datasets

  • Training Data: The authors generated a large synthetic dataset of approximately 280,000 "dirty" text images and their corresponding segmentation masks. This dataset was created by programmatically imposing artifacts onto clean images sourced from:
  • Printed Text: TextRecognitionDataGenerator (a synthetic dataset generator).
  • Handwritten Text: The IAM dataset (a widely used dataset for handwritten text recognition).
  • Evaluation Data:
    • Validation Set: A held-out portion of the generated synthetic dataset.
  • Real-World Test Data: The NIST SDB2 and NIST SDB6 datasets, which consist of scanned IRS tax return forms containing both printed and handwritten text with various artifacts.

Key Insights and Ideas

  • Weak Supervision is Key: The ability to generate a large, diverse training dataset automatically, without manual pixel-level annotation, is crucial for tackling the broad range of artifacts found in real documents. This significantly reduces the cost and effort of data preparation.
  • Artifacts are More Than Just Lines: The paper highlights that artifacts are not limited to simple lines but include boxes, smudges, and spurious strokes, requiring a robust segmentation approach.
  • Cleansing Improves OCR Dramatically: Removing artifacts before OCR processing leads to substantial improvements in recognition accuracy, demonstrating the practical value of artifact removal.
  • Generalization to Real-World Data: DeepErase shows promising performance even on out-of-distribution datasets like the IRS forms, suggesting its ability to generalize beyond the specific artifacts it was trained on.

Key Results

  • Segmentation Accuracy: DeepErase achieved segmentation errors of less than 5% on printed and handwritten text in their validation set, significantly outperforming traditional Hough transform methods and manually supervised CNNs.
  • OCR Accuracy Improvement (Validation Set):
    • For printed text, DeepErase-cleaned images improved Tesseract's recognition accuracy by 60.56% compared to using dirty images.
  • For handwritten text, DeepErase-cleaned images improved SimpleHTR's recognition accuracy by 31.20%.
  • OCR Accuracy Improvement (NIST IRS Dataset):
    • On printed IRS data, DeepErase lowered Tesseract's recognition error by 14.67%.
  • On handwritten IRS data, DeepErase lowered SimpleHTR's recognition error by 13.52%.
  • Qualitative Results: Visual inspection of the cleaned images showed them to be significantly more appealing and easier to read, with artifacts effectively removed.

Limitations and Future Work

  • Distribution Shift: The authors acknowledge that real-world data can exhibit distribution shifts, and while DeepErase performs well, there's room for improvement. Future work could focus on better modeling this test distribution during the artifact generation process.
  • Handwritten Text Accuracy: While improved, the recognition accuracy on handwritten IRS data still shows a relatively high error rate, partly attributed to distribution shift. Further research could explore more advanced techniques for handwritten text artifact removal.
  • Beyond Artifact Removal: The paper suggests that cleansed images could be beneficial for other downstream tasks beyond OCR, such as signature verification or document restoration, which could be explored further.

Summary updated

BibTeX
@inproceedings{qi2020deeperase,
  author = {Yike Qi and W. Ronny Huang and Qianqian Li and Jonathan Degange},
  title = {{DeepErase: Weakly Supervised Ink Artifact Removal in Document Text Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure