LOCR: Location-Guided Transformer for Optical Character Recognition

Yu Sun, Dongzhan Zhou, Chen Lin, Conghui He, Wanli Ouyang, Han-Sen ZhongEMNLP 2024 · DOI · OpenReview

TL;DR

This paper introduces LOCR, a novel transformer-based OCR model that leverages location information to significantly improve the accuracy and reduce repetitive errors in understanding complex academic documents. By integrating positional guidance into its autoregressive decoding process and introducing an interactive mode, LOCR achieves state-of-the-art performance and offers a more robust solution for digitizing challenging document layouts.

What Problem Does This Paper Address?

Academic documents, with their intricate mix of text, equations, tables, and figures, pose a significant challenge for Optical Character Recognition (OCR) systems. While end-to-end OCR models have advanced, they often struggle with repetitive outputs and accurately interpreting complex layouts, especially in "out-of-domain" documents (i.e., those with structures or content not seen during training). This leads to inaccurate transcriptions and a loss of valuable information. The authors highlight that existing transformer-based models, even those like Nougat, can get stuck in repetitive loops, failing to correctly identify and process the spatial relationships between different document elements.

Methods

The core innovation of LOCR lies in its location-guided transformer architecture. Unlike standard transformer models that primarily rely on sequential token information, LOCR integrates positional supervision directly into the autoregressive decoding process. This means that at each step, the model not only predicts the next token but also its corresponding location within the document. This is achieved through a prompt module that incorporates two-dimensional positional encodings and position detection heads. These heads, inspired by object detection algorithms like CenterNet, predict the bounding box of the next token.

Key methodological components include:

  • Positional Autoregression: LOCR simultaneously predicts the current token and the position of the next token, allowing it to learn the spatial context alongside the textual content.
  • Decay Strategies for Anti-Repetition: To combat repetitive outputs, LOCR employs "Accumulation Decay" and "Blank Decay" strategies during inference. Accumulation Decay penalizes grids that have already been heavily populated with tokens, while Blank Decay de-emphasizes empty or low-information grid areas.
  • Interactive OCR Mode: For challenging or out-of-domain documents, LOCR offers an interactive mode. Here, a human can provide a bounding box prompt to guide the model, allowing it to resume decoding from a specified location, significantly improving its ability to handle novel layouts.
  • Transformer Backbone: The model utilizes a Nougat-like structure as its backbone, combining a Swin Transformer image encoder with an mBART decoder.

Datasets

The authors introduce a novel, large-scale dataset for training and evaluating their model.

  • New Dataset: This dataset comprises over 77 million text-location pairs extracted from 125,000 academic document pages. A key differentiator is that its bounding boxes cover not only words but also mathematical symbols, tables, and various special formatting elements. The data was collected from academic papers sourced from arXiv.
  • Evaluation Datasets: The model is evaluated on a test set constructed from arXiv, as well as out-of-domain datasets of quantum physics and marketing documents.

Key Insights and Ideas

  • Spatial Context is Crucial for OCR Accuracy: The paper strongly demonstrates that simply processing text sequentially is insufficient for complex documents. Explicitly incorporating positional information is vital for accurate OCR, especially in avoiding repetitive errors.
  • Location Guidance Directly Addresses Repetition: By guiding the model's attention to specific locations, LOCR effectively prevents it from getting stuck in loops and hallucinating content. The decay strategies are a clever way to leverage this positional information for error reduction.
  • Interactive Mode Bridges the Gap for OOD Documents: The interactive feature is a pragmatic solution for handling the inherent variability and complexity of real-world documents, allowing human expertise to guide the model when it encounters novel or ambiguous layouts.
  • Comprehensive Bounding Boxes are Essential: The creation of a dataset with bounding boxes for mathematical symbols and other non-textual elements is a significant contribution, enabling more robust training for academic document understanding.

Key Results

LOCR significantly outperforms existing methods on the arXiv test set across multiple metrics, including edit distance, BLEU, METEOR, Precision, Recall, and F-measure.

  • Repetition Reduction: LOCR dramatically reduces repetition frequency:
  • From 4.4% to 0.5% on the arXiv dataset.
  • From 13.2% to 1.3% on OOD quantum physics documents.
  • From 8.1% to 1.8% on OOD marketing documents.
  • Performance Improvement: With the decay strategy tuned (σ = 0.85), LOCR achieves the best overall balance of high precision, recall, and F1 scores, along with the lowest edit distance and highest BLEU/METEOR scores on the arXiv test set.
  • Interactive Mode Effectiveness: The interactive mode successfully handles complex layouts and out-of-domain documents, as demonstrated by examples where human prompts corrected misinterpretations.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, transformer-based models are generally computationally intensive. The paper mentions training on multiple GPUs for extended periods.
  • Interactive Mode Reliance: For highly complex or novel documents, the interactive mode still requires human intervention, indicating that fully automated understanding of all document types remains a challenge.
  • Future Work: The authors suggest that their LOCR model and interactive tool can be applied to digitize documents across various fields with complex layouts, aiding academic research and LLM training. They also envision their dataset facilitating the development of better OCR models and enabling more robust fine-tuning for specific domain literature. The interactive semi-automatic data engine can also be used to construct datasets for enhancing model generalization.

Summary updated

Abstract

Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents.To tackle this issue, we propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on an original large-scale dataset comprising over 53M text-location pairs from 89K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms all existing methods in our test set constructed from arXiv.LOCR also eliminates repetition in the arXiv dataset, and reduces repetition frequency in OOD documents, from 13.19% to 0.04% for natural science documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.
BibTeX
@misc{sun2024locr,
  author = {Yu Sun and Dongzhan Zhou and Chen Lin and Conghui He and Wanli Ouyang and Han-Sen Zhong},
  title = {{LOCR: Location-Guided Transformer for Optical Character Recognition}},
  year = {2024},
  eprint = {2403.02127},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.02127},
}
Figure