Marior: Margin Removal and Iterative Content Rectification for Document Dewarping in the Wild

Jiaxin Zhang, Canjie Luo, Lianwen Jin, Fengjun Guo, Kai Ding

Abstract

Document dewarping in uncontrolled environments remains a significant challenge due to complex backgrounds, varying camera perspectives, and geometric distortions. Traditional models often require perfectly cropped inputs, failing when faced with "in the wild" captures. This paper introduces Marior, a novel framework that decouples the dewarping process into two distinct stages: Margin Removal and Iterative Content Rectification. By isolating the document from its background and iteratively refining the internal content, Marior achieves state-of-the-art performance in both geometric fidelity and optical character recognition (OCR) accuracy.

1. Introduction

Capturing documents via mobile devices often introduces significant distortions, including page curvature and perspective warping. Furthermore, real-world images frequently contain "marginal regions"—clutter such as desks, hands, or other background elements. While deep learning has advanced the field of document flattening, many existing architectures assume the input is already tightly cropped to the document boundaries. When these boundaries are missing or the background is noisy, standard models struggle to produce legible results.

Marior (Margin Removal and Iterative Content Rectification) addresses these limitations by treating margin removal and content flattening as separate, specialized tasks. This decoupling allows the system to handle uncropped or poorly framed images with high robustness.

2. Methodology

The Marior framework is composed of two primary modules designed to work in sequence.

2.1 Margin Removal Module (MRM)

The MRM focuses on isolating the document from the background. Unlike simple cropping tools, the MRM utilizes a semantic segmentation approach to identify the document's mask. To ensure the predicted mask adheres to the expected geometric regularity of a document (e.g., straight edges and quadrilateral-like forms), the authors employ a GAN-based framework. A discriminator enforces prior knowledge about document shapes, preventing the jagged or irregular masks common in standard segmentation. Once the mask is established, Thin Plate Spline (TPS) interpolation is used to perform a coarse, preliminary dewarp and crop.

2.2 Iterative Content Rectification Module (ICRM)

Following the initial crop, the image is processed by the ICRM to resolve internal distortions. This module predicts a "dense displacement flow," a pixel-level map indicating the necessary movement to align text lines horizontally and vertically.

  • Content-Aware Loss: A significant innovation in the ICRM is the use of a loss function that prioritizes the rectification of text and figures. By focusing the optimization on information-dense areas rather than blank margins, the model ensures higher legibility for OCR.
  • Adaptive Iteration: To achieve maximum smoothness, the ICRM can be applied multiple times. An intelligent stopping algorithm monitors the magnitude of changes between passes, halting the process once the rectification converges. This prevents over-processing and reduces computational waste.

3. Experimental Evaluation

The framework was trained primarily on the Doc3D dataset, a synthetic collection of 100,000 images. Evaluation was conducted using the DocUNet Benchmark, specifically comparing performance on both "Crop" (pre-cropped) and "Origin" (raw, uncropped) subsets. Additionally, the OCR_REAL dataset was utilized to measure the practical impact of dewarping on text recognition accuracy.

4. Results and Discussion

Marior demonstrates superior performance across multiple metrics, particularly on uncropped source images where previous methods often fail.

  • Geometric and Visual Fidelity: On the DocUNet benchmark, Marior achieved a Local Distortion (LD) score of 9.00 and a Multi-Scale Structural Similarity Index (MS-SSIM) of 0.4458. These results surpass previous leading models like DewarpNet and DocTr.
  • OCR Performance: The practical utility of the framework is most evident in the reduction of the Character Error Rate (CER). On raw "in the wild" images, the initial CER was 16.12%. After processing with Marior, the CER dropped to 4.18%, representing a significant leap in reliability for digital archiving and automated data entry.
  • Robustness: Qualitative analysis confirms that Marior remains effective even when document edges are partially cut off by the camera frame, a scenario that typically causes edge-based dewarping methods to collapse.

5. Conclusion

By decoupling margin removal from content rectification, Marior provides a robust solution for document dewarping in real-world conditions. The combination of GAN-based shape priors and iterative, content-aware refinement allows for high-quality flattening and significantly improved OCR results. Future work will explore end-to-end optimization of the MRM and ICRM modules to further enhance efficiency and performance in mobile applications.

Summary updated

BibTeX
@inproceedings{zhang2022marior,
  author = {Jiaxin Zhang and Canjie Luo and Lianwen Jin and Fengjun Guo and Kai Ding},
  title = {{Marior: Margin Removal and Iterative Content Rectification for Document Dewarping in the Wild}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=uJXFC0QoMQ},
}
Figure