TL;DR
This paper introduces DocMatcher, a novel document image dewarping method that leverages structural and textual lines from both the warped image and a template image to achieve state-of-the-art results. They also propose a new evaluation metric, mnCER, which better captures text readability and spatial positioning than existing metrics.
What Problem Does This Paper Address?
Digitizing physical documents, especially using smartphones, often results in distorted images due to uncontrolled environments and paper deformations like folds and creases. These distortions significantly hinder subsequent document analysis tasks such as text recognition and information extraction. While deep learning has improved dewarping, complex deformations and challenging conditions remain problematic. Existing methods often treat dewarping as a simple image-to-image mapping. Some recent approaches incorporate a "template" – a clean version of the expected document structure – to guide the dewarping process. However, these methods might implicitly leverage template information. This paper aims to explicitly use template information by focusing on structural and textual lines, which are fundamental to document layout.
Methods
DocMatcher employs a multi-stage approach:
- Background Removal: The authors fine-tune the Segment Anything Model (SAM) to create a binary mask, separating the document from the background.
- Line Detection: They adapt the LineFormer model to detect both structural and textual lines in the background-removed image. This involves extracting 2D line paths rather than mathematical functions.
- Pre-Unwarping: To simplify the dewarping task, the image is pre-unwarped using a homography transformation estimated by aligning detected lines. This step aims to axis-align all detected lines as much as possible.
- Template-Based Dewarping (Initial): The pre-unwarped image is then processed twice by the GeoTrTemplateLarge model, a state-of-the-art template-based dewarping method, to obtain an initial dewarped image.
- Line Matching: The core novelty lies here. Detected lines from the initially dewarped image are encoded (visual, positional, and optionally text features) and matched against lines from the template image using the LightGlue local feature matcher. False matches are filtered out.
- Line-Based Unwarping: The line correspondences are used to construct a dense transformation field (forward map). This involves:
- Projecting Matches: Densely sampling correspondences along matched lines to enforce pixel locations.
- Support Points: Adding points outside the convex hull of correspondences to improve interpolation and maintain straightness.
- Unmatched Lines: Projecting unmatched lines to enforce their straightness. A forward map is constructed, and a mean smoothing kernel is applied to the backward map to preserve text readability.
Datasets
- Inv3DReal: This dataset is used for both training and evaluation. It consists of 360 invoice document images with six deformation categories and three environment settings. The paper notes that it was introduced in a previous work [12].
Key Insights and Ideas
- Explicit Line Matching: Explicitly detecting and matching structural and textual lines between the warped image and a template provides a powerful way to leverage prior knowledge about document layout. This is more direct than implicit template usage.
- Robust Pre-Unwarping: Using detected lines for pre-unwarping, rather than just the document outline, makes the process more robust to imperfect outline detection.
- Novel Evaluation Metric (mnCER): Existing metrics like CER and ED are sensitive to text linearization and absolute positioning. The proposed
matched normalized Character Error Rate (mnCER)addresses this by matching words based on spatial locality and penalizing unmatched ground truth words, thus capturing both readability and 2D spatial accuracy. - Component Importance: The ablation study highlights that each component of DocMatcher (background removal, line detection, pre-unwarping, initial dewarping, and line-based unwarping) contributes to the overall performance, with line-based unwarping and the second GeoTrTemplateLarge inference showing significant impact on mnCER.
Key Results
- Quantitative: DocMatcher outperforms state-of-the-art methods on the Inv3DReal dataset across multiple metrics. Notably, it achieves:
- 32.6% improvement in Local Distortion (LD).
- 40.2% improvement in the proposed mnCER metric.
- Significant improvements in MS-SSIM, LPIPS, ED, and CER compared to GeoTrTemplateLarge.
- Qualitative: Visualizations show that DocMatcher produces dewarped images that are visually closer to the original, with improved positioning of text and visual elements, even under challenging conditions.
Limitations and Future Work
- Black Artifacts: The line-based unwarping stage can generate black artifacts at image borders when pixels are moved towards the center and no external pixels are available to fill the gap. The paper mentions experimenting with filling these artifacts, which slightly impacts metrics like LD but maintains overall performance.
- Future Work: The authors suggest integrating more local features, such as points of interest, to further improve correspondence-based unwarping.
Summary updated