DavarOCR: A Toolbox for OCR and Multi-Modal Document Understanding

Liang Qiao, Hui Jiang, Ying Chen, Can Li, Pengfei Li, Zaisheng Li, Baorui Zou, Dashan Guo, Yingda Xu, Yunlu Xu, Zhanzhan Cheng, Yi NiuarXiv 2022 · arXiv

TL;DR

DavarOCR is an open-source, PyTorch-based toolbox designed to bridge the gap between basic text recognition and advanced document understanding. By introducing a highly modular architecture and a unified data format, it supports 19 state-of-the-art algorithms across 9 distinct tasks, making it easier than ever to build multi-modal deep learning pipelines for complex document analysis.

What Problem Does This Paper Address?

Traditional open-source OCR toolboxes focus heavily on perception—simply detecting bounding boxes and recognizing text characters. However, real-world applications require cognition: understanding the layout, determining the reading order, recognizing complex tables, and extracting key entities (like prices or dates from an invoice).

While toolboxes like MMOCR or early versions of PaddleOCR excel at basic text detection and recognition, they offer limited or fragmented support for downstream document understanding tasks. DavarOCR addresses this gap by providing a unified, extensible framework that seamlessly integrates both basic OCR and multi-modal document intelligence.

Table 1: Comparison of support algorithms between different open-source OCR toolboxes. All methods in Basic OCR Tasks only use visual information. In Document Understanding tasks, methods labelled with ♥means using visual features, with ♣ means using textual features, with ♦means using positional features.

Methods

DavarOCR inherits its core modular philosophy from mmdetection and mmcv, ensuring full compatibility with the MMOCR ecosystem. To support complex document understanding, the authors extended the standard deep learning pipeline (Backbone, Neck, ROI Extractor, Head) by introducing three specialized module types:

  1. Transformation: Learnable deformation correction modules (like Affine or TPS) to rectify warped or irregular text before recognition.
  2. Embedding: Textual and 2D positional embedding layers that allow models to encode semantic meaning and spatial relationships simultaneously.
  3. Connect: Intermediate layers designed for feature enhancement and multi-modal fusion (e.g., Bi-LSTM layers or cross-modality fusion mechanisms).

This modularity allows researchers to easily mix and match components across different tasks. For example, the end-to-end Key Information Extraction (KIE) model, TRIE, simultaneously shares features across text detection, recognition, and information extraction branches.

Figure 1: The modular design for the end-to-end KIE model TRIE.

Additionally, DavarOCR introduces a Uniform Data Label Format. Instead of managing disparate annotation styles for different tasks, it uses a standardized JSON structure capable of handling multi-level annotations (such as page-level blocks vs. individual text lines) seamlessly.

Figure 2: Illustration of the basic data label

Datasets

The toolbox was evaluated using prominent benchmarks in the document intelligence space:

  • WildReceipt: Used for evaluating Key Information Extraction (KIE).
  • PubLayNet: Used for evaluating document layout analysis.

Key Insights and Ideas

  • The Multi-Modal Multiplier: Purely visual models struggle with document semantics. By integrating textual and positional embeddings directly into the visual pipeline, models gain a holistic understanding of the document structure.
  • Cross-Task Module Sharing: A text rectification module built for scene text recognition can be reused directly in video text spotting or KIE pipelines without rewriting the underlying architecture.
  • Unified Data Pipelines: Standardizing the data format means that data loading, augmentation, and formatting functions are shared across completely different domains, drastically reducing engineering overhead.

Key Results

The authors conducted extensive ablation studies to demonstrate the impact of multi-modal integration within DavarOCR:

  • Key Information Extraction (KIE): On the WildReceipt dataset, upgrading the TRIE model from a purely visual modality (78.25% F1-Score) to a multi-modal approach combining Visual, Textual, and Positional features boosted the F1-Score to 87.08%.
Table 2: Multi-modal ablation results of the Key Informa- tion Extraction task on WildReceipt[27].
  • Layout Analysis: Evaluating the VSR model on the PubLayNet dataset showed that incorporating character-level text, sentence-level text, and positional features alongside visual data pushed the mean Average Precision (mAP) from 92.6% up to 95.8%.
Table 3: Multi-modal ablation results of the Layout Analysis task on PubLayNet[36]. (C) means Character-level textual feature, and (S) means Sentence-level textual feature.

Limitations and Future Work

While DavarOCR implements 10 advanced algorithms not found in any other open-source toolbox, the authors note that it is an ongoing project. The current release lays the groundwork for layout analysis, table recognition, and reading order detection, but full support for Document Visual Question Answering (DocVQA) and deeper Table Understanding models are deferred to future updates.

Summary updated

Abstract

This paper presents DavarOCR, an open-source toolbox for OCR and document understanding tasks. DavarOCR currently implements 19 advanced algorithms, covering 9 different task forms. DavarOCR provides detailed usage instructions and the trained models for each algorithm. Compared with the previous opensource OCR toolbox, DavarOCR has relatively more complete support for the sub-tasks of the cutting-edge technology of document understanding. In order to promote the development and application of OCR technology in academia and industry, we pay more attention to the use of modules that different sub-domains of technology can share. DavarOCR is publicly released at https://github.com/hikopensource/Davar-Lab-OCR.
BibTeX
@misc{qiao2022davarocr,
  author = {Liang Qiao and Hui Jiang and Ying Chen and Can Li and Pengfei Li and Zaisheng Li and Baorui Zou and Dashan Guo and Yingda Xu and Yunlu Xu and Zhanzhan Cheng and Yi Niu},
  title = {{DavarOCR: A Toolbox for OCR and Multi-Modal Document Understanding}},
  year = {2022},
  eprint = {2207.06695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.06695},
}
Figure