Enhanced Optical Character Recognition by Optical Sensor Combined with BERT and Cosine Similarity Scoring (Student Abstract)

Woohyeon Moon, Sarvar Hussain Nengroo, Taeyoung Kim, Jihui Lee, Seungah Son, Dongsoo HarAAAI 2024 · DOI

TL;DR

This paper tackles the challenge of noisy optical character recognition (OCR) by proposing a novel hybrid approach. It combines traditional OCR with BERT and cosine similarity scoring, dynamically switching between methods based on image clarity to significantly boost accuracy on degraded text.

What Problem Does This Paper Address?

Optical Character Recognition (OCR) is a fundamental technology for converting images of text into machine-readable data. However, its performance degrades significantly when images are "noisy" – meaning they contain artifacts, distortions, or low quality that obscure the characters. This paper addresses the limitations of conventional OCR systems in handling such noisy inputs, aiming to improve their robustness and accuracy.

Methods

The proposed method, dubbed BERTOCR, introduces a multi-stage approach:

  1. Initial OCR: A standard OCR model processes the input image to extract text and provide an initial accuracy probability.
  2. Confidence-Based Decision: A hyperparameter K (set to 0.5 in this work) is used to determine if the initial OCR output is "good" (probability > K) or "potentially erroneous" (probability <= K).
  3. BERT for Contextual Refinement: If the OCR output is deemed potentially erroneous, the text is fed into a BERT model. BERT is used to predict masked tokens (words) within the text, leveraging its understanding of language context to correct errors. This is done by masking a potentially noisy word and having BERT predict it.
  4. Cosine Similarity for Image Alignment: To further refine the predictions and account for visual similarity, cosine similarity scoring is employed. Words predicted by BERT with high probability are converted back into text images. These predicted images are then compared to the original noisy words' images using cosine similarity.
  5. Final Score and Selection: A confidence rate from BERT and the image similarity score from cosine similarity are multiplied. The final output is determined by selecting the option with the highest score using an argmax function.

The novelty lies in the dynamic integration of BERT and cosine similarity scoring, triggered only when the initial OCR output is uncertain, and the explicit incorporation of image similarity alongside contextual language understanding.

Datasets

The paper evaluates its model on three datasets:

  • Wikipedia: 7.8 million sentences.
  • CNN dailymail: 300 thousand sentences.
  • Fiction: 2000 sentences.

Artificial noise was introduced to these datasets to simulate real-world noisy image scenarios.

Key Insights and Ideas

  • Dynamic Hybridization is Key: The core idea is that not all OCR tasks require complex NLP. By dynamically switching to BERT and cosine similarity only for noisy inputs, the model achieves efficiency and improved accuracy.
  • Context and Visuals Together: BERT excels at understanding language context, but it doesn't inherently consider visual similarity. Adding cosine similarity bridges this gap, ensuring that the corrected text not only makes linguistic sense but also visually aligns with the original (albeit noisy) image.
  • Noise is a Major Bottleneck: The significant performance improvement highlights how much conventional OCR struggles with even moderate levels of noise, underscoring the need for more robust methods.

Key Results

The proposed BERTOCR model demonstrated substantial improvements over conventional OCR:

  • Conventional OCR Average Accuracy: 0.0594
  • OCR + BERT Average Accuracy: 0.2486
  • BERTOCR Average Accuracy: 0.2576

The BERTOCR model outperforms the conventional OCR model by an impressive factor of 4.3367 times. It also showed a 3.6% improvement over the OCR + BERT model.

Limitations and Future Work

  • Computational Cost: While the paper doesn't explicitly detail computational costs, the addition of BERT and cosine similarity scoring for noisy inputs likely increases processing time compared to a purely conventional OCR approach. The dynamic switching mechanism aims to mitigate this.
  • Hyperparameter Tuning: The effectiveness of the K threshold for determining noise levels might require fine-tuning for different types or levels of noise.
  • Future Work: The authors suggest that this work opens possibilities for further advancements in OCR, particularly in challenging noisy input scenarios, by leveraging NLP models. This implies exploring more sophisticated NLP models or different integration strategies.

Summary updated

BibTeX
@inproceedings{moon2024enhanced,
  author = {Woohyeon Moon and Sarvar Hussain Nengroo and Taeyoung Kim and Jihui Lee and Seungah Son and Dongsoo Har},
  title = {{Enhanced Optical Character Recognition by Optical Sensor Combined with BERT and Cosine Similarity Scoring (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30483},
}
Figure