Dewarping Document Image by Displacement Flow Estimation with Fully Convolutional Network

Guo-Wang Xie, Fei Yin, Xu-Yao Zhang, Cheng-Lin Liu

TL;DR

This paper introduces a novel framework for dewarping document images captured by cameras, which often suffer from distortions and background noise. By estimating pixel-wise displacements using a fully convolutional network (FCN) and incorporating a Local Smooth Constraint (LSC), the method effectively rectifies images while finely removing backgrounds, achieving state-of-the-art performance in preserving local details.

What Problem Does This Paper Address?

In today's world, we increasingly rely on smartphone cameras to capture documents, leading to a proliferation of distorted images. Unlike flatbed scanners, camera-based document images are prone to various deformations due to factors like uneven lighting, perspective shifts, paper folding, and wrinkling. These distortions significantly hinder the performance of document recognition systems. While existing dewarping methods exist, they often struggle with complex distortions, require specialized hardware, or make strong assumptions that limit their generality. This paper aims to address these limitations by proposing a robust and generalizable approach that can effectively rectify distorted document images and simultaneously clean up the background.

Methods

The core of the proposed approach is a Fully Convolutional Network (FCN) designed to estimate pixel-wise displacements. This "displacement flow" essentially describes how each pixel in the distorted image needs to move to achieve its correct position in a rectified, flat image. The FCN is trained to regress these ground-truth displacements from synthesized distorted documents.

A key innovation is the introduction of a Local Smooth Constraint (LSC). This regularization term is added to the loss function during training to ensure that the estimated displacements are smooth and continuous within local regions. This helps to prevent artifacts and preserve fine details in the rectified image, addressing a common issue where distortions might be smoothed out too aggressively.

The FCN also has a second output branch for foreground/background classification. This allows the system to not only rectify the document content but also to effectively remove or isolate the background. The rectification process then involves mapping the foreground pixels to their new positions in the rectified image using interpolation, guided by the predicted displacements. Delaunay triangulation is used for this interpolation to handle scattered pixels.

Datasets

The paper primarily uses a synthetically generated dataset for training. This dataset consists of 80,000 images created by warping scanned documents and applying various distortion functions (folds, curves) and augmenting them with background textures and jitter. The authors emphasize that their synthetic dataset is designed to be different from existing ones, aiming for better generalization to real-world images.

For evaluation, the authors use the Ma et al. [15] benchmark dataset, which contains real-world distorted document images.

Key Insights and Ideas

  • Displacement Flow as the Core Representation: Viewing document dewarping as estimating a pixel-wise displacement field is a powerful paradigm that allows for a unified approach to handling complex distortions.
  • Local Smooth Constraint (LSC) for Detail Preservation: The LSC is crucial for achieving high-quality results. It ensures that the estimated displacements are locally consistent, preventing the introduction of artifacts and preserving the fine details of the document. This is a significant improvement over methods that might over-smooth or introduce unwanted distortions.
  • Dual-Task FCN for Rectification and Background Removal: The FCN's ability to simultaneously predict displacements and classify foreground/background is an efficient way to handle both aspects of document image processing in a single network.
  • Generalization from Synthetically Generated Data: The paper demonstrates that a model trained on a carefully synthesized dataset, even if significantly different from real-world data, can achieve strong generalization performance. This highlights the importance of robust data generation strategies.

Key Results

The proposed method achieves state-of-the-art performance on the Ma et al. [15] benchmark dataset, particularly in terms of local details.

  • Quantitative Results: The paper reports a Local Distortion (LD) score of 8.50, which is superior to previous methods like DocUnet (14.08) and DewarpNet (8.98/8.95). While the MS-SSIM (global similarity) score is competitive but not the absolute best, the authors argue their focus is on expanding distortions rather than strictly matching scanned images.
  • Qualitative Results: Visual comparisons (Figure 7 and 8) show that the method effectively rectifies images with various distortions, producing visually pleasing results with clean backgrounds. The LSC is shown to significantly improve the smoothness and quality of the rectified image (Figure 5).
  • Efficiency: The approach is described as easy to implement and consumes moderate computing resources, processing a 1024x960 image in approximately 0.67 to 0.72 seconds.

Limitations and Future Work

  • Edge Artifacts: The authors acknowledge that the edges of partially rectified images are "still not neat enough."
  • Computational Speed: While efficient, the speed of calculation could be further improved.
  • Future Work: The authors plan to enhance performance and eliminate the need for post-processing steps. They also aim to further improve the speed of calculation.

Summary updated

Abstract

As camera-based documents are increasingly used, the rectification of distorted document images becomes a need to improve the recognition performance. In this paper, we propose a novel framework for both rectifying distorted document image and removing background finely, by estimating pixel-wise displacements using a fully convolutional network (FCN). The document image is rectified by transformation according to the displacements of pixels. The FCN is trained by regressing displacements of synthesized distorted documents, and to control the smoothness of displacements, we propose a Local Smooth Constraint (LSC) in regularization. Our approach is easy to implement and consumes moderate computing resource. Experiments proved that our approach can dewarp document images effectively under various geometric distortions, and has achieved the state-of-the-art performance in terms of local details and overall effect.
BibTeX
@inproceedings{xie2020dewarping,
  author = {Guo-Wang Xie and Fei Yin and Xu-Yao Zhang and Cheng-Lin Liu},
  title = {{Dewarping Document Image by Displacement Flow Estimation with Fully Convolutional Network}},
  booktitle = {DAS},
  year = {2020},
  url = {https://openreview.net/forum?id=m3V68hrl-F},
}
Figure