DTrOCR: Decoder-Only Transformer for Optical Character Recognition

Masato FujitakeWACV 2024

TL;DR

This paper introduces DTrOCR, a novel decoder-only Transformer architecture for optical character recognition (OCR). By leveraging a pre-trained generative language model (GPT) and eliminating the need for a separate vision encoder, DTrOCR achieves state-of-the-art performance across various text recognition tasks, including scene text, printed text, and handwritten text, in both English and Chinese.

What Problem Does This Paper Address?

Traditional OCR systems often rely on an encoder-decoder architecture. The encoder's role is to extract visual features from an image, which are then fed to a decoder to generate the text sequence. While effective, this approach can be computationally expensive and complex. Furthermore, while language models (LMs) have shown great success in natural language processing (NLP), their direct application to OCR, especially in a decoder-only fashion without a dedicated vision encoder, remained underexplored. This paper aims to simplify the OCR pipeline by demonstrating that a powerful generative LM, when adapted, can directly process image features and perform text recognition effectively, even outperforming existing encoder-decoder methods.

Methods

The core innovation of DTrOCR lies in its decoder-only Transformer architecture. Instead of a traditional encoder-decoder setup, DTrOCR takes an image as input and directly feeds it into a decoder module.

Here's a breakdown of the key components:

  • Patch Embedding Module: The input image is first divided into fixed-size patches. These patches are then transformed into vectors and combined with positional encodings to create a sequence that can be processed by the Transformer decoder. This essentially converts the visual information into a format suitable for the language model.
  • Decoder Module: This module is based on a pre-trained generative language model (specifically, GPT). It receives the patch-embedded image sequence and a special [SEP] token to signal the start of text generation. The decoder then autoregressively predicts the text sequence, token by token, until it generates an [EOS] token. The use of a generative LM allows the model to leverage its learned linguistic knowledge for more accurate text prediction.
  • Pre-training and Fine-tuning: The model is first pre-trained on large synthetic datasets to bridge the gap between visual and language information. This is followed by fine-tuning on specific real-world datasets for various OCR tasks.

Datasets

The paper utilizes a combination of synthetic and real-world datasets for training and evaluation:

  • Pre-training:
    • English: PILE dataset.
  • Chinese: CC100, and the Chinese NLP Corpus.
  • Synthetic text images were generated using SynthTIGER and Text Render.
  • Fine-tuning and Evaluation:
    • Scene Text Recognition: IIIT5k, SVT, IC13, IC15, SVTP, CUTE, SROIE Task 2.
  • Handwritten Text Recognition: IAM Handwriting Database.
  • Chinese Text Recognition (CTR): A large CTR benchmark dataset comprising scene, web, document, and handwriting subsets.
  • Receipt OCR: Scanned Receipts OCR and Information Extraction (SROIE) Task 2.

Key Insights and Ideas

  • Encoder is Not Always Necessary: The most significant insight is that a dedicated vision encoder is not essential for achieving high accuracy in text recognition. A powerful decoder-only generative LM can directly process image features and learn to recognize text effectively.
  • Generative LMs Excel in OCR: The success of DTrOCR highlights the potential of generative LMs, pre-trained on vast text corpora, to be adapted for visual tasks like OCR. Their inherent understanding of language structure and context proves highly beneficial.
  • Simplicity and Effectiveness: By removing the encoder and complex cross-attention mechanisms, DTrOCR offers a simpler architecture that is both computationally efficient and highly accurate.
  • Pre-training is Crucial: The paper emphasizes the importance of pre-training on large synthetic datasets to connect visual and linguistic information, enabling the model to generalize better to real-world scenarios.

Key Results

DTrOCR consistently outperformed existing state-of-the-art methods across various benchmarks:

  • Scene Text Recognition: Achieved superior accuracy on both synthetic and real datasets, outperforming methods like TrOCR, ABINet, and PARSeq. For instance, on the CUTE dataset with real data, DTrOCR achieved 99.1% word accuracy, compared to 98.3% for PARSeq.
  • SROIE Task 2 (Receipt OCR): Outperformed existing methods with a word-level F1 score of 98.37%, significantly higher than TrOCR's 96.58%.
  • IAM Handwriting Database: Achieved a character error rate (CER) of 2.38%, outperforming TrOCR (2.89%) and other methods.
  • Chinese Text Recognition (CTR): Demonstrated strong performance across all subsets, achieving 89.7% word accuracy on the combined benchmark, surpassing TrOCR (79.3%) and MaskOCRLARGE (67.9%).
  • Efficiency: While not the primary focus, the paper notes that previous studies had low reported processing speeds (FPS), and their work, based on generative LMs, aims to improve this.

Limitations and Future Work

  • Model Size and Training Data: The paper notes that larger pre-trained decoder models tend to achieve higher accuracy. This suggests that further improvements could be made by using even larger LMs or more extensive pre-training data.
  • Computational Cost of LMs: While DTrOCR simplifies the architecture, the computational cost associated with large generative LMs can still be a factor. The authors mention that further improvements in LM efficiency techniques could benefit their approach.
  • Future Work: The authors suggest that applying their decoder-only approach to other vision-language tasks and exploring more efficient LM adaptation techniques are promising directions. They also believe their work opens up new avenues for research in text recognition by challenging the traditional encoder-decoder paradigm.

Summary updated

BibTeX
@inproceedings{fujitake2024dtrocr,
  author = {Masato Fujitake},
  title = {{DTrOCR: Decoder-Only Transformer for Optical Character Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure