NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement

Giordano Cicchetti, Danilo ComminielloarXiv 2024 · DOI · arXiv

Document digitization is often a complex process. Between shaky camera captures, ancient fading ink, and environmental noise like coffee stains or watermarks, Optical Character Recognition (OCR) systems frequently fail to produce accurate results. While Diffusion Probabilistic Models (DPMs) have emerged as a gold standard for image restoration, they are typically slow and prone to "hallucinating" or substituting characters—a critical flaw when textual accuracy is paramount.

In their paper "NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement," Giordano Cicchetti and Danilo Comminiello introduce a framework designed to make diffusion both fast enough for practical use and accurate enough to satisfy demanding OCR requirements.

The Perception-Distortion Tradeoff in Document Enhancement

Traditional document enhancement faces a significant "perception-distortion" tradeoff. Generative models, such as GANs, can produce visually pleasing results that inadvertently distort underlying text, leading to high OCR error rates. While standard diffusion models offer superior generative quality, they suffer from two major hurdles:

  1. Inference Latency: Standard DPMs require hundreds or thousands of iterations to generate an image, making them impractical for industrial-scale document processing.
  2. Character Integrity: Because DPMs are designed for diversity, they can occasionally "swap" similar-looking characters (e.g., turning an 'e' into an 'o'), compromising the utility of the restored document.

The NAF-DPM Methodology

The authors propose a multi-stage architecture to address these challenges:

  • Predict-and-Refine Strategy: Instead of initiating the process from pure noise, the system utilizes an Initial Predictor (a scaled-down NAFNet) to restore low-frequency information, such as the general document structure. The DPM Denoiser then focuses solely on predicting the "residual"—the high-frequency details and sharp edges.
  • NAFNet Backbone: Both the predictor and denoiser utilize a Nonlinear Activation-Free (NAF) architecture. By replacing traditional activation functions (like ReLU or Sigmoid) with simple multiplication-based gates, the model achieves significantly higher computational efficiency.
  • Fast ODE Sampling: To mitigate latency, the framework employs a high-order DPM-solver. This allows the model to converge on high-quality results in just 10 to 20 iterations, a substantial improvement over the 50+ iterations required by previous state-of-the-art models like DocDiff.
  • OCR-Guided Finetuning: To ensure character integrity, the authors integrated a differentiable CRNN (Convolutional Recurrent Neural Network) module during training. This module "reads" the restored text and applies a Connectionist Temporal Classification (CTC) loss, penalizing the model if the restored pixels do not match the ground-truth characters.

Experimental Evaluation and Datasets

The NAF-DPM model was evaluated across several benchmark datasets representing diverse document restoration tasks:

  • Deblurring: The BMVC OCR Test Dataset, featuring 66,000 blurry images for training and a specialized 94-image set for OCR testing.
  • Binarization: The annual DIBCO (Document Image Binarization Competition) datasets from 2017, 2018 (H-DIBCO), and the challenging 2019 set, which includes ancient papyrus documents.
Overview of the datasets and tasks used for evaluation, including deblurring and binarization of printed and handwritten documents.

Key Results and Insights

NAF-DPM outperformed existing CNN, GAN, and DPM-based methods across multiple metrics:

  • Superior Deblurring: On the BMVC dataset, NAF-DPM achieved a PSNR of 34.377, a significant improvement over the previous diffusion leader, DocDiff (29.787).
  • Reduced OCR Errors: NAF-DPM reduced the Character Error Rate (CER) to 1.55% (and 1.40% with finetuning), nearly halving the 2.78% error rate of DocDiff.
  • Binarization Performance: The model established a new state-of-the-art on the DIBCO 2019 dataset, demonstrating robustness even on highly textured, ancient substrates.
Visual comparison of degraded input images versus the restored outputs generated by the NAF-DPM framework.

The success of the NAF-DPM framework highlights several key insights. First, frequency separation is vital; by allowing a simple network to handle the "big picture" while the diffusion model manages fine details, the generative process becomes more stable. Second, the NAF architecture suggests that traditional non-linear activations may be unnecessary for document tasks, as simple gating mechanisms provide sufficient representational power with lower latency. Finally, the OCR-based loss function acts as a semantic anchor, ensuring the model preserves linguistic information rather than just producing a "clean" image.

Conclusion and Future Directions

While NAF-DPM represents a significant advancement, future work may focus on reducing training complexity and developing a unified framework capable of handling multiple forms of degradation (e.g., stains, folds, and fading) simultaneously. Nevertheless, by combining speed with character-level accuracy, NAF-DPM provides a viable path for the practical application of diffusion models in document digitization.

Summary updated

Abstract

Real-world documents may suffer various forms of degradation, often resulting in lower accuracy in optical character recognition (OCR) systems. Therefore, a crucial preprocessing step is essential to eliminate noise while preserving text and key features of documents. In this paper, we propose NAF-DPM, a novel generative framework based on a diffusion probabilistic model (DPM) designed to restore the original quality of degraded documents. While DPMs are recognized for their high-quality generated images, they are also known for their large inference time. To mitigate this problem we provide the DPM with an efficient nonlinear activation-free (NAF) network and we employ as a sampler a fast solver of ordinary differential equations, which can converge in a few iterations. To better preserve text characters, we introduce an additional differentiable module based on convolutional recurrent neural networks, simulating the behavior of an OCR system during training. Experiments conducted on various datasets showcase the superiority of our approach, achieving state-of-the-art performance in terms of pixel-level and perceptual similarity metrics. Furthermore, the results demonstrate a notable character error reduction made by OCR systems when transcribing real-world document images enhanced by our framework. Code and pre-trained models are available at https://github.com/ispamm/NAF-DPM.
BibTeX
@misc{cicchetti2024naf,
  author = {Giordano Cicchetti and Danilo Comminiello},
  title = {{NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement}},
  year = {2024},
  eprint = {2404.05669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05669},
}
Figure