GraDeT-HTR: A Resource-Efficient Bengali Handwritten Text Recognition System utilizing Grapheme-based Tokenizer and Decoder-only Transformer

Md. Mahmudul Hasan, Ahmed Nesar Tahsin Choudhury, Md. Mosaddek KhanEMNLP 2025

TL;DR

This paper introduces GraDeT-HTR, a novel and resource-efficient system for Bengali Handwritten Text Recognition (HTR). By employing a decoder-only Transformer architecture combined with a grapheme-based tokenizer specifically designed for Bengali script, the system achieves state-of-the-art performance on multiple benchmarks, addressing the significant gap in HTR for this widely spoken language.

What Problem Does This Paper Address?

Despite Bengali being the sixth most spoken language globally, robust handwritten text recognition (HTR) systems for it are severely lacking. This deficiency stems from the inherent complexities of the Bengali script, which features intricate ligatures, numerous diacritics, and significant allographic variations. Compounding these linguistic challenges is a scarcity of large, high-quality annotated datasets. Existing HTR approaches often rely on encoder-decoder architectures, but the application of Transformer-based models, particularly decoder-only variants, to Bengali HTR remains largely unexplored. This paper aims to bridge this gap by developing an effective and efficient HTR system for Bengali.

Methods

The GraDeT-HTR system is built upon a decoder-only Transformer architecture, a departure from the more common encoder-decoder models in OCR. This architecture, inspired by models like DTrOCR, processes image patches directly through a series of Transformer decoder blocks. A key innovation is the integration of a grapheme-based tokenizer, specifically the "BnGraphemizer," which is tailored to the unique visual units of the Bengali script. Unlike traditional subword tokenizers (like BPE) that can be unstable with handwritten variations, the grapheme tokenizer enforces a one-to-one mapping between visually coherent units and model output tokens, aiming to improve recognition accuracy. The system also incorporates a text detection module (BN-DRISHTI) to segment full-page images into word-level inputs for the recognition module. The training strategy involves a two-stage process: pre-training on a large synthetic dataset (4.5 million line-level and 7 million word-level samples) and then fine-tuning on real, human-annotated Bengali handwriting datasets.

Datasets

The paper utilizes a combination of synthetic and real-world datasets:

  • Synthetic Data: Generated using tools adapted from the VRD framework, comprising approximately 4.5 million line-level samples and 7 million word-level samples. This data is used for pre-training.
  • Real Human-Annotated Datasets (for fine-tuning):
    • BN-HTRd
    • BanglaWriting
    • BanglaLekha-Isolated
    • IIIT-Indic
    • ICDAR 2024
    • Bongabdo

These datasets collectively provide over 338,000 handwritten word images for fine-tuning the word-level models.

Key Insights and Ideas

  • Grapheme-based Tokenization is Crucial for Bengali: The study empirically demonstrates that a grapheme-based tokenizer significantly outperforms traditional subword (BPE) tokenization for Bengali HTR. This is attributed to the script's complex ligatures and diacritics, where subword units can be visually unstable and lead to errors.
  • Decoder-Only Transformers are Viable for Bengali HTR: The paper shows that a decoder-only Transformer architecture, trained from scratch without relying on large pre-trained Bengali language models, can achieve state-of-the-art performance. This offers a practical path for low-resource settings.
  • Word-Level Recognition Outperforms Line-Level: Models trained on word-level images achieve better performance than those trained on line-level images. This suggests that finer granularity in input processing leads to more accurate transcription.
  • Synthetic Data Pre-training is Effective: The two-stage pre-training on synthetic data, first on line-level and then on word-level images, effectively prepares the model for fine-tuning on limited real-world data.

Key Results

  • The proposed Word_BnG_Random model (word-level input, grapheme tokenizer, randomly initialized decoder) achieved the lowest Character Error Rate (CER) and Word Error Rate (WER) on the BN-HTRd and Bongabdo datasets, outperforming other configurations and existing OCR systems.
  • Specifically, the model achieved 6.19% CER and 14.20% WER on BN-HTRd and 8.68% CER and 23.56% WER on Bongabdo.
  • The grapheme-based tokenizer not only improved accuracy but also reduced model size by approximately 45% (from 162M to 87M parameters) compared to BPE tokenization with a pretrained LLM.
  • The system demonstrated faster inference speeds (9.95 words/s) compared to a larger variant (8.30 words/s), highlighting its efficiency.
  • The end-to-end pipeline's performance is sensitive to the quality of the text detection module, indicating that improvements in word segmentation can further boost recognition accuracy.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Limited Training Data Diversity: The training data primarily consists of handwritten text on white backgrounds, which may limit adaptability to more varied real-world scenarios (e.g., different backgrounds, noise).
  • Scarcity of Pretrained Bengali LLMs: The experiments were constrained by the lack of extensive pretrained GPT-2 models for Bengali, limiting the breadth of exploration with decoder initialization.

Future work suggestions include:

  • Expanding the synthetic dataset to include diverse backgrounds and noise artifacts.
  • Pre-training autoregressive language models on extensive Bengali text.
  • Experimenting with other autoregressive language models (e.g., Llama).
  • Refining the text detection module to reduce segmentation errors.

The authors are publicly releasing their complete pipeline under the MIT license to support future research in Bengali HTR.

Summary updated

BibTeX
@inproceedings{hasan2025gradet,
  author = {Md. Mahmudul Hasan and Ahmed Nesar Tahsin Choudhury and Mahmudul Hasan and Md. Mosaddek Khan},
  title = {{GraDeT-HTR: A Resource-Efficient Bengali Handwritten Text Recognition System utilizing Grapheme-based Tokenizer and Decoder-only Transformer}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure