EfficientOCR: An Extensible, Open-Source Package for Efficiently Digitizing World Knowledge

Tom Bryan, Jacob Carlson, Abhishek Arora, Melissa DellEMNLP 2023

TL;DR

This paper introduces EffOCR, a novel open-source OCR package designed for efficient and cost-effective digitization of large-scale document collections. By reframing OCR as an image retrieval problem rather than a sequence-to-sequence task, EffOCR achieves remarkable sample efficiency, making it feasible to train and deploy models for low-resource languages and diverse document types.

What Problem Does This Paper Address?

Billions of public domain documents are currently inaccessible to modern NLP tools because they exist only in hard copy or have low-quality digital versions. Digitizing these vast archives is a significant challenge due to the sheer volume of data, the diversity of languages and scripts, and the high cost of existing OCR solutions. Current state-of-the-art OCR models, often based on sequence-to-sequence architectures, require massive datasets and computational resources for training, making them impractical for large-scale, low-resource applications. This limits our ability to index, retrieve, and analyze historical texts, and to incorporate them into language model training.

Methods

EffOCR introduces a novel approach to OCR by modeling it as a word or character level image retrieval problem, departing from the prevalent sequence-to-sequence (seq2seq) paradigm.

  • Localization: EffOCR first uses object detection models (like YOLO and MMDetection) to identify the locations of lines, words, and characters within an image.
  • Recognition: Instead of predicting characters sequentially, EffOCR learns to recognize characters and words by comparing their image embeddings to a pre-computed offline dictionary of exemplar embeddings. This means the model learns the visual appearance of characters/words rather than their sequential dependencies.
  • Sample Efficiency: This retrieval-based approach is significantly more sample-efficient because it doesn't require learning complex language models. The model only needs to learn the visual characteristics of characters and words.
  • Extensibility and Deployment: The package is designed for ease of use, offering an off-the-shelf toolkit, a repository of pre-trained models, and tools for efficient model training and customization. It also supports ONNX runtime for fast deployment.

Datasets

  • Training: The paper mentions training EffOCR on digital fonts and a modest number of character and word crops from real-world documents. For specific benchmarks, they used labeled sets of newspaper lines and table cells.
  • Evaluation:
    • 20 million historical U.S. newspaper scans: Digitized to create the "American Stories" dataset.
  • U.S. National Archives documents: Evaluated for zero-shot performance.
  • Historical Japanese document collection: Used to demonstrate performance on a challenging, low-resource scenario where other OCR solutions failed.
  • Chronicling America collection: Used for scalability testing.

Key Insights and Ideas

  • Reframing OCR as Retrieval: The core insight is that treating OCR as an image retrieval problem, rather than a seq2seq task, dramatically improves sample efficiency. This is because learning visual similarity is less data-intensive than learning sequential language generation.
  • Democratizing OCR: EffOCR's sample efficiency and ease of use make advanced OCR accessible to researchers and institutions with limited budgets and deep learning expertise, especially for low-resource languages.
  • Cost-Effectiveness at Scale: The architecture and ONNX integration enable significantly cheaper deployment for large-scale digitization compared to commercial solutions or even other open-source seq2seq models.
  • Zero-Shot and Few-Shot Capabilities: The retrieval-based approach allows for surprisingly good performance even with very limited or no specific training data for a particular document type or language.

Key Results

  • Cost Savings: Digitizing 20 million newspaper pages cost $60K USD with EffOCR, whereas Google Cloud Vision would have cost over $23 million USD.
  • Accuracy:
    • Achieved 11.2% CER on a zero-shot test of U.S. National Archive documents (trained only on newspapers).
  • Achieved 0.7% CER on historical Japanese firm-level records, outperforming Baidu OCR which mispredicted over half of characters.
  • Demonstrated viable few-shot performance with CERs as low as 5% (Japanese) and 7% (English) using only 5% of the training data.
  • Speed: EffOCR (Small) demonstrated the fastest inference speed among tested frameworks in a CPU environment. ONNX integration provides up to a 3x speedup on CPU.

Limitations and Future Work

  • Illegible Documents: EffOCR, being vision-only, may struggle with documents where large portions are illegible, and language understanding might be necessary.
  • High-Resource Languages & No Cost Constraints: For English and similar high-resource languages where cost is not a barrier, commercial products like Google Cloud Vision might offer better performance.
  • Handwriting Support: EffOCR currently does not support handwriting, though plans are in place to expand the model zoo with crowd-sourced annotations and synthetic handwriting generators.
  • Model Zoo Expansion: The authors plan to expand the model zoo to include more languages and scripts, including handwriting, through crowd-sourcing efforts.
  • Integration with Layout Parser: EffOCR is being integrated with Layout Parser to enable combined document layout analysis and OCR.

Summary updated

BibTeX
@inproceedings{bryan2023efficientocr,
  author = {Tom Bryan and Jacob Carlson and Abhishek Arora and Melissa Dell},
  title = {{EfficientOCR: An Extensible, Open-Source Package for Efficiently Digitizing World Knowledge}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}

Loading related content...

Figure