A Novel Pipeline for Improving Optical Character Recognition through Post-processing Using Natural Language Processing

Aishik Rakshit, Samyak Mehta, Anirban Dasgupta

TL;DR

While standard Optical Character Recognition (OCR) excels at reading license plates, it often stumbles over the nuances of handwriting and complex fonts. This paper introduces an end-to-end pipeline that combines computer vision for text extraction with state-of-the-art NLP models (like BART and Alpaca-LoRA) to "clean up" OCR errors, drastically reducing character and word error rates.

What Problem Does This Paper Address?

OCR technology is the backbone of document digitization, but it has a "last mile" problem. While state-of-the-art models handle printed, high-contrast text well, they struggle with handwritten notes, varied font styles, and similar-looking characters (like the letter 'o' vs. the number '0').

When an OCR model produces "noisy" text, it creates a domino effect: downstream tasks like text summarization, sentiment analysis, and Named Entity Recognition (NER) fail because the input data is corrupted. The authors argue that instead of trying to build a "perfect" vision model, we should use the context-awareness of Natural Language Processing (NLP) to correct the mistakes vision models inevitably make.

Methods

The authors propose a two-module pipeline:

Module A: The OCR Engine

  1. Segmentation: Uses an A* Path Planning algorithm to find the optimal paths between lines of text, effectively slicing a multi-line document into individual strips.
  2. Classification: A modified DenseNet-121 architecture classifies each line as either "printed" or "handwritten." This allows the system to route the image to the most appropriate OCR model.
  3. Recognition: The authors compared two heavyweights: TrOCR (a Transformer-based encoder-decoder) and PP-OCR (Paddle-Paddle OCR).

Module B: The NLP Post-Processor

This is the core innovation. Once the raw text is extracted (and likely contains errors), it is passed through an NLP model to be "translated" into corrected English. They tested three approaches:

  • ByT5: A Google model that processes raw bytes instead of tokens, making it highly resilient to character-level noise.
  • BART: A denoising autoencoder specifically designed to reconstruct corrupted text.
  • Alpaca-LoRA: A fine-tuned version of Meta’s LLaMA 7B model. The authors used a simple prompt: "Fix all the errors in the sentence: [text]."

To train these, they used synthetic data generation, taking clean text and intentionally injecting "OCR-like" errors (swapping similar characters, deleting spaces) to teach the models how to fix them.

Datasets

The study utilized a diverse array of benchmarks to test different "real-world" conditions:

  • Born-Digital Images: Text from digital sources (web/email).
  • Incidental Scene Text: Text found in the wild (signs, labels).
  • License Plate Dataset: High-contrast, standardized printed text.
  • Single Line Handwritten Text: Specifically for testing handwriting recognition.
  • Synthetic OSCAR Corpus: Used to train the NLP correction models by simulating OCR degradation.

Key Insights and Ideas

  • Vision vs. Language: The paper highlights that OCR is a multi-modal problem. Vision identifies the shapes, but Language provides the context to know that "H0use" should actually be "House."
  • The Power of LLMs for Error Correction: One of the most striking findings was how effectively Large Language Models (LLMs) like Alpaca-LoRA could "hallucinate" the correct version of a word based on surrounding context, even when the OCR output was significantly garbled.
  • Segmentation Matters: By using A* path planning for segmentation, the model handles the physical layout of the page more robustly than simple horizontal projections, which often fail when lines of text overlap slightly.

Key Results

The results demonstrate that NLP post-processing isn't just a minor tweak—it's a game-changer:

  • Handwriting Accuracy: On the Kaggle Handwritten dataset, the raw OCR output had a Character Error Rate (CER) of 0.169. After applying Alpaca-LoRA post-processing, the CER plummeted to 0.023.
  • Word Error Rate (WER): On synthetic datasets, the baseline WER was 0.455. The BART model reduced this to 0.062, and Alpaca-LoRA brought it down further to 0.045.
  • Model Comparison: The authors found that TrOCR generally outperformed PP-OCR for handwriting (CER of 0.09 vs 0.53), while PP-OCR was superior for structured text like license plates.

Limitations and Future Work

  • Skewed Documents: The current segmentation module assumes the document is relatively straight. If the page is rotated or the handwriting is heavily slanted, the A* path planning may struggle.
  • Computational Overhead: Running a 7B parameter model like Alpaca-LoRA for post-processing is computationally expensive compared to the OCR step itself.
  • Future Directions: The authors suggest developing more robust segmentation for "unconstrained" or messy documents and further optimizing the NLP models for speed to allow for real-time correction.

Summary updated

Abstract

Optical Character Recognition (OCR) technology finds applications in digitizing books and unstructured documents, along with applications in other domains such as mobility statistics, law enforcement, traffic, security systems, etc. The state-of-the-art methods work well with the OCR with printed text on license plates, shop names, etc. However, applications such as printed textbooks and handwritten texts have limited accuracy with existing techniques. The reason may be attributed to similar-looking characters and variations in handwritten characters. Since these issues are challenging to address with OCR technologies exclusively, we propose a post-processing approach using Natural Language Processing (NLP) tools. This work presents an end-to-end pipeline that first performs OCR on the handwritten or printed text and then improves its accuracy using NLP.
BibTeX
@misc{rakshit2023novel,
  author = {Aishik Rakshit and Samyak Mehta and Anirban Dasgupta},
  title = {{A Novel Pipeline for Improving Optical Character Recognition through Post-processing Using Natural Language Processing}},
  year = {2023},
  eprint = {2307.04245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.04245},
}
Figure