Recognition-based Vs Syntax-directed Models for Numerical Field Extraction in Handwritten Documents

Clément Chatelain, Laurent Heutte, Thierry Paquet2014

TL;DR

This paper tackles the challenge of extracting numerical information (like zip codes or customer IDs) from messy, unconstrained handwritten documents. The authors propose and compare two distinct strategies: one that prioritizes recognition first, and another that leverages syntax rules upfront. Their findings suggest that a syntax-directed approach significantly outperforms a recognition-first method for this task.

What Problem Does This Paper Address?

In today's world, vast amounts of information are exchanged digitally and on paper. While extracting data from structured forms or machine-printed documents is relatively well-understood, processing free-layout, unconstrained handwritten documents remains a significant hurdle. This is particularly true for incoming mail, where the location and structure of information can vary wildly. The paper focuses on the specific problem of extracting numerical fields (like zip codes, customer codes, and phone numbers) from these challenging documents. Unlike forms where fields have predefined locations, here the system must simultaneously figure out where the numerical data is and what it is, a classic information extraction challenge.

Methods

The authors propose two main strategies for numerical field extraction:

  • Recognition-based Strategy: This approach follows a more traditional pipeline for handwriting recognition. It begins by segmenting the document into individual components and attempting to recognize each one as a digit. Then, it uses syntactic rules to identify potential numerical fields from these recognized digits. The core idea is to recognize first, then locate.
  • Syntax-directed Strategy: This method flips the order. It first uses a "low-level" model that incorporates the syntactic rules of the target numerical fields (e.g., the number of digits in a zip code) and a very coarse classification of components (e.g., digit-like, text-like). This allows it to localize potential numerical fields before attempting detailed recognition. Only the localized fields are then subjected to a more focused recognition process. This strategy prioritizes syntax and localization upfront.

Both strategies involve stages of segmentation, recognition, and identification, but the order and emphasis differ significantly.

Datasets

The paper evaluates its proposed methods on a "real incoming mail document database." This database consists of 293 documents each for learning and testing. The documents contain zip codes, phone numbers, and customer codes, all of which are labeled with their position and numerical values. The authors note that this database contains private information and is unfortunately unavailable for public diffusion.

Key Insights and Ideas

  • The "Sayre's Paradox" in Handwritten Document Analysis: The paper highlights the inherent difficulty in handwritten document processing, where segmentation (breaking down the image) and recognition (identifying characters) are mutually dependent. You need to segment to recognize, but you also need to recognize to know what to segment.
  • Syntax is King for Unconstrained Data: The most striking insight is that for unconstrained handwritten documents, leveraging the syntax of the target numerical fields (their structure, number of digits, separators) upfront is far more effective than trying to recognize everything first.
  • Recognition-First Can Be Overwhelmed: The recognition-based approach, while intuitive, suffers from generating too many potential hypotheses due to the inherent variability and noise in handwriting. This makes it difficult to filter and locate the correct information accurately.
  • Focused Recognition is More Efficient: The syntax-directed approach, by localizing potential fields first, significantly reduces the search space for the recognition stage. This leads to a more constrained and ultimately more successful recognition problem.

Key Results

The paper presents a recall-precision trade-off graph comparing the two approaches. The syntax-directed strategy consistently outperforms the recognition-based strategy across different "TOPn" thresholds (where n represents the number of best alignment paths considered).

  • For the recognition-based approach, the best performance achieved was around 57% recall with 20% precision (at TOP1).
  • The syntax-directed approach achieved recall below 50% at TOP1 but with a higher precision rate, and generally dominated the trade-off curve, indicating better overall performance.

The authors conclude that the syntax-directed approach is more robust and promising for this task.

Limitations and Future Work

The authors acknowledge that while their proposed strategies are generic and industrializable, they rely on the syntax of the numerical fields being constrained enough. They also mention that many questions remain open, including:

  • Segmentation and Recognition of Both Numerical and Textual Components: Further improvements are needed in handling the complex interplay between recognizing numerical data and distinguishing it from textual noise.
  • Conception of Efficient Models: Developing even more sophisticated models for both numerical and non-numerical components is an ongoing challenge.

For future work, they plan to combine the syntax-directed method with a keyword spotting system, aiming to improve overall system reliability. They also suggest that the syntax-directed method could be applied to other types of numerical fields and languages, provided their syntax is sufficiently constrained.

Summary updated

Abstract

rouen.fr In this article, two different strategies are proposed for numerical field extraction in weakly constrained hand-written documents. The first extends classical handwrit-ing recognition methods, while the second is inspired from approaches usually chosen in the field of information ex-traction from electronic documents. The models and the implementation of these two opposed strategies are de-scribed, and experimental results on a real handwritten mail database are presented.
BibTeX
@misc{chatelain2014recognition,
  author = {Clément Chatelain and Laurent Heutte and Thierry Paquet},
  title = {{Recognition-based Vs Syntax-directed Models for Numerical Field Extraction in Handwritten Documents}},
  year = {2014},
}
Figure