DocBinFormer: A Two-Level Transformer Network for Effective Document Image Binarization

Risab Biswas, Swalpa Kumar Roy, Ning Wang, Umapada Pal, Guang-Bin HuangarXiv 2023 · OpenReview

TL;DR

This paper introduces DocBinFormer, a novel two-level Vision Transformer (ViT) architecture for document image binarization. By effectively capturing both global and local image features through a hierarchical attention mechanism, DocBinFormer achieves state-of-the-art performance on challenging degraded document datasets, outperforming existing methods.

What Problem Does This Paper Address?

Document image binarization is a critical preprocessing step for various document analysis tasks like OCR and information extraction. However, real-world documents often suffer from significant degradation (e.g., aging, ink bleed, uneven lighting, artifacts), making it challenging to accurately separate foreground text from the background. Traditional methods struggle with diverse degradations, and while CNNs are good at local features, they can miss long-range dependencies. Existing ViT models often focus on global features, potentially neglecting crucial local details. This paper aims to bridge this gap by developing a more robust and effective binarization method that can handle complex degradations by leveraging both global context and fine-grained local information.

Methods

The core innovation is the DocBinFormer, a two-level Vision Transformer (TL-ViT) architecture. Unlike standard ViTs that process images as fixed-size patches, DocBinFormer introduces a hierarchical approach:

  • Patch and Sub-Patch Tokenization: The input image is first divided into larger patches, and then each patch is further divided into smaller sub-patches.
  • Two-Level Transformer Encoder:
    • Global Transformer Encoder: Processes the larger patches to capture global contextual information.
  • Local Transformer Encoder: Processes the sub-patches to capture fine-grained local details.
  • Multi-Head Self-Attention: Both the global and local encoders utilize multi-head self-attention mechanisms to learn dependencies within patches and sub-patches, respectively. This allows the model to adaptively focus on relevant image regions.
  • Feature Fusion: The outputs from the global and local encoders are combined to create a rich feature representation that encapsulates both global structure and local texture.
  • Transformer Decoder: A standard transformer decoder then reconstructs the binarized output image from the fused features.

Crucially, DocBinFormer achieves this without relying on convolutional layers, processing image data directly through attention mechanisms.

Datasets

The authors trained and evaluated their model on several standard benchmark datasets for document image binarization:

  • DIBCO (Document Image Binarization Contest) datasets: DIBCO 2009, 2010, 2011, 2012, 2013, 2014, 2016, 2017, 2018, and 2019.
  • H-DIBCO (Handwritten Document Image Binarization Contest) datasets: Used in conjunction with DIBCO datasets.

The evaluation used a leave-one-out cross-validation strategy, where one year's dataset was used for testing, and the rest for training.

Key Insights and Ideas

  • Hierarchical Attention is Key: The paper strongly advocates for a hierarchical approach to feature extraction in document binarization. Simply using global attention (like in standard ViTs) or local attention (like in CNNs) is insufficient. Combining them effectively is crucial.
  • Sub-Patch Level Detail Matters: The authors highlight that breaking down patches into sub-patches and applying local attention is vital for capturing intricate details that are easily missed by larger receptive fields or global attention alone. This is particularly important for handling variations in text size, font, and subtle degradations.
  • Transformer-Native Approach: By avoiding convolutional layers and relying solely on transformer blocks for both global and local feature extraction, DocBinFormer presents a more unified and potentially more powerful architecture for this task.
  • Adaptability of Attention: The dynamic nature of attention mechanisms allows them to adapt their focus based on the specific content of the input, making them more robust to diverse degradations compared to fixed receptive fields in CNNs.

Key Results

DocBinFormer demonstrated superior performance across multiple metrics (PSNR, FM, Fps, DRD) on the DIBCO and H-DIBCO datasets.

  • Quantitative: The paper reports that DocBinFormer outperforms state-of-the-art methods on most DIBCO datasets. For instance, it achieved the highest PSNR on DIBCO 2009, 2010, 2011, 2012, 2016, and 2017, often by significant margins (e.g., 0.85% and 3.98% absolute improvement on PSNR for DIBCO 2009 compared to the previous best). It was also highly competitive on other datasets.
  • Qualitative: Visual results presented in the paper (Figures 2-10) show DocBinFormer producing cleaner binarized images with fewer artifacts and better preservation of text details, especially in severely degraded samples, compared to other methods. Red bounding boxes highlight areas where other models failed, while blue boxes highlight DocBinFormer's successful recovery.

Limitations and Future Work

  • Computational Cost: While the paper mentions that creating very small sub-patches (e.g., 2x2) can be computationally expensive, they found a balance with 8x8 sub-patches. Further exploration of hyperparameter tuning, especially patch and sub-patch sizes, could be beneficial.
  • Ablation Study: The ablation study in Section IV-E suggests that higher embedding dimensions generally improve results, but the impact of other factors like patch size and layer configurations is explored. The paper found a reasonable balance between performance and computational complexity with their chosen configuration.
  • Future Work: The authors suggest that further experimentation with hyperparameter variations and potentially exploring different transformer architectures could lead to even greater improvements. The paper also hints at the potential for this architecture in other low-level vision tasks beyond binarization.

Summary updated

Abstract

In real life, various degradation scenarios exist that might damage document images, making it harder to recognize and analyze them, thus binarization is a fundamental and crucial step for achieving the most optimal performance in any document analysis task. We propose DocBinFormer (Document Binarization Transformer), a novel two-level vision transformer (TL-ViT) architecture based on vision transformers for effective document image binarization. The presented architecture employs a two-level transformer encoder to effectively capture both global and local feature representation from the input images. These complimentary bi-level features are exploited for efficient document image binarization, resulting in improved results for system-generated as well as handwritten document images in a comprehensive approach. With the absence of convolutional layers, the transformer encoder uses the pixel patches and sub-patches along with their positional information to operate directly on them, while the decoder generates a clean (binarized) output image from the latent representation of the patches. Instead of using a simple vision transformer block to extract information from the image patches, the proposed architecture uses two transformer blocks for greater coverage of the extracted feature space on a global and local scale. The encoded feature representation is used by the decoder block to generate the corresponding binarized output. Extensive experiments on a variety of DIBCO and H-DIBCO benchmarks show that the proposed model outperforms state-of-the-art techniques on four metrics. The source code will be made available at https://github.com/RisabBiswas/DocBinFormer.
BibTeX
@misc{biswas2023docbinformer,
  author = {Risab Biswas and Swalpa Kumar Roy and Ning Wang and Umapada Pal and Guang-Bin Huang},
  title = {{DocBinFormer: A Two-Level Transformer Network for Effective Document Image Binarization}},
  year = {2023},
  eprint = {2312.03568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.03568},
}
Figure