Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, Muhammad Abdul-MageedACL 2024

TL;DR

This paper introduces "Qalam," a novel multimodal LLM designed for Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR). Qalam achieves state-of-the-art performance by combining a SwinV2 encoder with a RoBERTa decoder, significantly outperforming existing methods on diverse Arabic datasets.

What Problem Does This Paper Address?

Arabic script presents unique challenges for OCR and HWR due to its cursive nature, context-dependent characters, ligatures, dots, and the critical role of diacritics. Existing OCR systems often struggle with these complexities, and the availability of comprehensive, diverse annotated datasets for Arabic has been a bottleneck. This paper aims to address these challenges by developing a robust, high-performing model specifically for Arabic script recognition.

Methods

The core of Qalam is a Vision Encoder-Decoder (VED) framework.

  • Encoder: A SwinV2 Transformer is used as the vision encoder. This model is chosen for its proficiency in handling high-resolution inputs and capturing rich spatial information. It's further pre-trained on a large corpus of Arabic manuscripts.
  • Decoder: A RoBERTa model serves as the language decoder. It's pre-trained using Masked Language Modeling on diverse Arabic datasets, including Arabic Wikipedia and AraC4, and is modified to handle longer sequences and understand diacritics.
  • Training: The model is trained using a combination of real and synthetically augmented data. Data augmentation involves adding variations in fonts to existing samples. The training procedure utilizes the Adam optimizer with a cosine learning rate scheduler.

Datasets

The paper introduces MIDAD, a comprehensive benchmark for Arabic OCR and HWR, which aggregates several existing datasets:

  • MADBase: 60k training and 10k testing images of Arabic handwritten digits.
  • AHCD: 16k samples of handwritten Arabic characters.
  • ADAB: 15k samples of Tunisian town and village names in Arabic handwriting.
  • Alexuw: 25k samples of 109 different Arabic words.
  • Shotor: 120k grayscale images of various Farsi phrases.
  • PATS-A01: 22k line images of printed Arabic text from classic literature.
  • IDPL-PFOD: 30k synthetic TIF images of printed Farsi text lines with various distortions.
  • UPTI: 10k synthetic Urdu text lines in Nastaliq font.
  • OnlineKHATT: 10k Arabic text lines from 623 authors, stored in InkML format and converted to images.
  • KHATT: 1,000 handwritten Arabic forms from diverse writers.
  • Historical Manuscripts: 120 images from historical Arabic manuscripts.

A synthetic dataset of 60k image-text pairs with over 28 diverse fonts was also created for fine-tuning.

Key Insights and Ideas

  • SwinV2 + RoBERTa Synergy: The combination of SwinV2's visual processing capabilities and RoBERTa's linguistic understanding, particularly its ability to handle diacritics, is key to Qalam's success.
  • Importance of Diacritics: The paper emphasizes that diacritics are crucial for Arabic script meaning and pronunciation, and Qalam's ability to process them is a significant advantage over models that omit them.
  • Pre-training on Arabic Data: Pre-training on large-scale Arabic manuscript and text data is vital for achieving high performance, highlighting the limitations of models primarily trained on English.
  • Handling High-Resolution Inputs: Qalam, especially with the SwinV2 encoder, demonstrates superior handling of high-resolution inputs, a common limitation in other models.

Key Results

Qalam achieves state-of-the-art performance on Arabic OCR and HWR tasks:

  • Word Error Rate (WER): 0.80% for HWR tasks and 1.18% for OCR tasks.
  • Midad Score: 0.94% (a novel evaluation metric combining HWR and OCR performance).
  • On specific datasets like MADBase and AHCD, Qalam achieves near-perfect recognition (0.01 WER).
  • It shows strong performance on line recognition tasks, achieving a WER of 3.95% on OnlineKHATT.
  • In "in the wild" scenarios (KHATT dataset), Qalam achieved a Character Error Rate (CER) of 10.43%, significantly improving upon the SoTA of 18.45%.

Limitations and Future Work

  • Diacritics Handling: While Qalam handles diacritics well, ARBERT (a decoder tested) struggled when diacritics were not in its vocabulary. This highlights the ongoing challenge.
  • Code-Switching and Dialects: The model may struggle with code-switching and dialectal variations common in real-world Arabic writing.
  • Scene Text Recognition (STR) and Full-Page Recognition: Qalam currently shows limitations in these more complex tasks.
  • Generalization: Qalam is specifically designed for Arabic. Its effectiveness on other scripts without further fine-tuning is not guaranteed.
  • Future Work: The authors suggest exploring further pre-training strategies and adapting the model for other complex scripts. The development of the MIDAD benchmark is also intended to facilitate future research.

Summary updated

BibTeX
@inproceedings{bhatia2024qalam,
  author = {Gagan Bhatia and El Moatez Billah Nagoudi and Fakhraddin Alwajih and Muhammad Abdul-Mageed},
  title = {{Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure