Generative Pipeline for Data Augmentation of Unconstrained Document Images with Structural and Textural Degradation (Student Abstract)

Arnab Poddar, Abhishek Kumar Sah, Soumyadeep Dey, Pratik Jawanpuria, Jayanta Mukhopadhyay, Prabir Kumar BiswasAAAI 2023 · DOI

Abstract

Researchers have developed a three-stage generative pipeline that creates realistic, "messy" synthetic handwritten document images. By simulating both structural deformations (like strike-outs) and textural degradations (like aging and shadows), the system provides a much-needed dataset with perfect ground-truth labels for training and evaluating robust OCR and document analysis tools.

Introduction: The Challenge of Real-World Documents

Most Document Image Understanding (DIU) systems—the technology behind OCR and digital archiving—are trained on "clean" data: neatly scanned pages with clear text. However, real-world documents, especially those captured via mobile devices, are often "messy." They suffer from textural degradation (blurring, uneven lighting, smudging, and aging) and structural deformation (scribbles, strike-outs, and underlines).

Examples of real-world document degradation including textural noise (shadows, aging) and structural noise (receipt formatting).

The primary bottleneck in developing robust models is the lack of large-scale "dirty" datasets with perfect, pixel-level annotations. Without such data, it is difficult to train deep learning models to distinguish between relevant text and background noise like coffee stains or strike-outs.

Methodology: A Three-Stage Generative Framework

The proposed framework bridges the data gap using three distinct modules:

  1. Generator-1 (Handwriting Generation): This module converts raw text into handwritten images on a clean background, extending the "GANwriting" framework to ensure stylistic diversity.
  2. Generator-2 (Structural Deformation): This stage introduces structural noise. Using a pool of 3,000 real-world stroke templates, the system randomly adjusts position, rotation, and size to simulate realistic strike-outs and underlines.
  3. Generator-3 (Texture & Background): Finally, a pix2pix-based architecture with ResNet blocks applies textures mimicking aging, smudging, and non-linear illumination (shadows).
Synthetic structural deformation: a strike-out applied to the word 'reason'.

By decoupling these steps, the pipeline generates images while simultaneously outputting "free" ground-truth labels for both text content and foreground/background pixel segmentation.

Experimental Results and Key Insights

The researchers evaluated the impact of these degradations using a standard CRNN OCR model on the IAM dataset. The results highlight a significant performance drop as noise complexity increases:

  • Baseline (Clean IAM): 10.12% Character Error Rate (CER).
  • Textural Degradation Only: 24.63% CER.
  • Structural Deformation Added: 45.11% CER.

These findings suggest that structural noise is significantly more damaging to recognition accuracy than textural noise. The ability to decouple these failure modes allows researchers to pinpoint exactly where a system fails—whether it's due to the appearance of the paper or physical obstructions like strike-outs.

Conclusion and Future Work

This generative approach provides a scalable way to create high-quality training data for document enhancement models. While currently focused on English and simple strike-out templates, future work aims to incorporate more complex scribbling patterns and extend the framework to diverse scripts (e.g., Arabic or CJK). This research facilitates the development of the next generation of OCR systems capable of handling truly unconstrained, real-world document images.

Summary updated

BibTeX
@inproceedings{poddar2023generative,
  author = {Arnab Poddar and Abhishek Kumar Sah and Soumyadeep Dey and Pratik Jawanpuria and Jayanta Mukhopadhyay and Prabir Kumar Biswas},
  title = {{Generative Pipeline for Data Augmentation of Unconstrained Document Images with Structural and Textural Degradation (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27009},
}
Figure