DocNLC: A Document Image Enhancement Framework with Normalized and Latent Contrastive Representation for Multiple Degradations

Ruilu Wang, Yang Xue, Lianwen Jin

TL;DR

This paper introduces DocNLC, a novel framework for document image enhancement that tackles multiple degradation types simultaneously. By employing a contrastive learning approach with both direct and latent representations, DocNLC learns to disentangle common features across various degradations, leading to improved performance and generalization compared to methods that focus on single degradation types.

What Problem Does This Paper Address?

Document images, especially those captured by cameras, are often plagued by a variety of degradations. These can include watermarks, blurring, noise, uneven lighting, and ink bleed. Existing deep learning-based solutions typically focus on addressing only one or a few of these issues at a time. This specialization makes them less practical for real-world scenarios where documents can exhibit multiple degradations simultaneously. Furthermore, training models on datasets with only single degradation types can lead to poor generalization and an inability to handle diverse real-world inputs. The core challenge is to develop a unified framework that can effectively enhance documents with multiple, often ambiguous, degradations.

Methods

DocNLC introduces a contrastive learning paradigm designed for document image enhancement. The framework comprises three main components:

  • Degradation Normalization and Compensation (DNC) Module: This module first normalizes the features of degraded document images using instance normalization to align them. Crucially, it then compensates for any information lost during normalization by integrating the original, unprocessed features. This ensures that the restoration module receives complete information.
  • Restoration Module: A standard U-Net-like architecture is used for the pixel-level correction and remapping of the document image.
  • Latent Contrastive Module: This is the core innovation. It employs a contrastive learning strategy that considers both direct (pixel-level) and latent (feature-level) representations. Unlike typical contrastive learning methods that aim to maximize inter-class distances, DocNLC focuses on explicitly decorrelating representations of different degradation types. This is achieved by minimizing redundancy between features from various degradations while preserving essential content information. This module is used only during training to tune parameters and does not add complexity to the final inference network.

The contrastive learning approach is designed to be asymmetric, assigning different importance to primary and other modalities, and aims to create degradation-independent representations.

Datasets

The paper utilizes publicly available datasets for training and evaluation.

  • Training: The authors created a unified dataset by merging several publicly available datasets and artificially generating missing degradation types for each original document sample. This resulted in samples with five degradation types: backgrounds, watermarks, shadows, noise, and blur. The DIBCO 2008-2019 dataset series was used for generating these augmented training samples.
  • Evaluation: Four publicly available independent datasets from previous DIBCO competition series were used for evaluation. These datasets represent a range of document types and degradations.

Key Insights and Ideas

  • Unified Enhancement is Possible: The paper demonstrates that a single framework can effectively handle multiple degradation types, moving beyond specialized models.
  • Contrastive Learning for Degradation Disentanglement: The key insight is that contrastive learning can be adapted to learn representations that are invariant to different degradation types, rather than just distinguishing between classes.
  • Latent Contrast is Crucial: Explicitly learning relationships between latent representations of different degradations is vital for achieving robust enhancement and minimizing redundancy.
  • Normalization with Compensation: Simply normalizing features can lead to information loss. The DNC module's compensation step is critical for preserving necessary details for accurate restoration.
  • Primary Modality Matters: The choice of the "primary modality" (the degradation type used as the reference in contrastive learning) significantly impacts performance. Degradations with higher mutual information with the ground truth (like background) lead to better results.

Key Results

  • State-of-the-Art Performance: DocNLC outperforms existing state-of-the-art methods on four DIBCO datasets in both pre-training and fine-tuning stages, achieving the best average PSNR and SSIM scores.
  • Generalization: The pre-trained DocNLC model shows strong generalization capabilities across different datasets, even without fine-tuning.
  • Parameter Efficiency: DocNLC achieves its performance gains with a minimal increase in parameters compared to baseline models.
  • Qualitative Improvements: Visualizations show DocNLC effectively removing shadows, backgrounds, watermarks, and noise, significantly improving document readability.

Limitations and Future Work

  • Heavily Blurred Images: The paper acknowledges that DocNLC performs poorly on heavily blurred document images or those with multi-layered shadows, leading to inconsistent character colors. This is identified as a limitation and an avenue for future research.
  • Primary Modality Choice: While "background" was found to be the best primary modality, the paper suggests that judicious selection of this modality is crucial for optimal performance.
  • Future Directions: The authors suggest exploring more advanced contrastive learning strategies and further investigating the impact of different primary modalities.

Summary updated

Abstract

Document Image Enhancement (DIE) remains challenging due to the prevalence of multiple degradations in document images captured by cameras. In this paper, we respond an interesting question: can the performance of pre-trained models and downstream DIE models be improved if they are bootstrapped using different degradation types of the same semantic samples and their high-dimensional features with ambiguous inter-class distance? To this end, we propose an effective contrastive learning paradigm for DIE — a Document image enhancement framework with Normalization and Latent Contrast (DocNLC). While existing DIE methods focus on eliminating one type of degradation, DocNLC considers the relationship between different types of degradation while utilizing both direct and latent contrasts to constrain content consistency, thus achieving a unified treatment of multiple types of degradation. Specifically, we devise a latent contrastive learning module to enforce explicit decorrelation of the normalized representations of different degradation types and to minimize the redundancy between them. Comprehensive experiments show that our method outperforms state-of-the-art DIE models in both pre-training and fine-tuning stages on four publicly available independent datasets. In addition, we discuss the potential benefits of DocNLC for downstream tasks. Our code is released at https://github.com/RylonW/DocNLC
BibTeX
@inproceedings{wang2024docnlc,
  author = {Ruilu Wang and Yang Xue and Lianwen Jin},
  title = {{DocNLC: A Document Image Enhancement Framework with Normalized and Latent Contrastive Representation for Multiple Degradations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28366},
}

Loading related content...

Figure