Alpha-numerical sequences extraction in handwritten documents

Thomas Simon, Clément Chatelain, Laurent Heutte2015

TL;DR

This paper introduces a novel system for extracting alpha-numerical sequences (like keywords and numerical fields) from unconstrained handwritten documents. Instead of trying to recognize the entire document, it focuses on a "global line model" that distinguishes between relevant information and noise, enabling efficient extraction even from messy handwriting.

What Problem Does This Paper Address?

The paper tackles the challenge of information extraction from unconstrained handwritten documents. While electronic communication has grown, a vast amount of important information is still conveyed through handwritten documents. Existing handwriting recognition systems often struggle with the variability of handwriting, the lack of fixed layouts, and the sheer computational cost of full document recognition. This makes it difficult to automatically extract specific pieces of information, such as keywords or numerical data, which are crucial for tasks like document categorization or data entry. The authors aim to bridge this gap by developing a system that can efficiently and reliably extract targeted information without needing to perfectly recognize every character.

Methods

The core of the proposed system is a global handwriting line model. This model takes a holistic approach to processing text lines, rather than focusing on individual words or characters. It's built upon two key components:

  • Relevant Information Model: This part is designed to capture specific sequences of interest, such as keywords or numerical fields. These are modeled using Hidden Markov Models (HMMs), a standard technique in sequence recognition, where each character is represented by its own HMM.
  • Irrelevant Information Model (Shallow Parsing): This component acts as a filter, modeling everything else in the text line that isn't considered relevant. This includes out-of-vocabulary words, punctuation, spaces, and general noise. This is handled by a "shallow language model" whose distributions are learned from training data.

The system operates by treating a text line as a competition between these two models. A hyperparameter, G, controls the balance between prioritizing relevant information (favoring recall) and irrelevant information (favoring precision). This allows for flexible tuning based on application needs. The authors also detail preprocessing steps like binarization, line segmentation, and skew/slant correction, followed by feature extraction using a sliding window approach and HMM decoding.

Datasets

  • RIMES database: This public database of French incoming mails was used for both training and evaluation. It contains 1150 documents from different writers. 950 documents were used for learning the model parameters, and 200 documents were used for testing.

Key Insights and Ideas

  • Global Line Model: The shift from word-level or character-level processing to a global line model is a key innovation. This allows the system to handle the inherent variability of handwriting more effectively by considering the entire line's context.
  • Relevant vs. Irrelevant Information Dichotomy: Explicitly modeling both what is and what is not relevant information within a line is a powerful strategy. This allows the system to be more robust to noise and less constrained by the need for perfect recognition.
  • Shallow Parsing for Noise Reduction: Using a shallow parsing model for irrelevant information is an efficient way to filter out noise and focus on the target data, avoiding the computational burden of full document understanding.
  • Tunable Recall/Precision: The introduction of the G hyperparameter provides a mechanism to easily adjust the system's trade-off between finding all relevant information (recall) and ensuring that what is found is indeed relevant (precision), making it adaptable to different use cases.

Key Results

The paper demonstrates the effectiveness of their approach on the RIMES database.

  • Keyword Extraction: The system achieved the best performance for keyword extraction, with a break-even point of 0.6 on the Recall/Precision curve.
  • Numerical Field Extraction: Numerical field extraction yielded a break-even point of 0.4. The authors attribute this to the difficulty of detecting numerical information using baseline-dependent features.
  • Lexicon Size Impact: Increasing the lexicon size (the set of target sequences) generally decreased performance but the system remained robust, showing only a gradual decline. Even with a lexicon of 100 words, performance was comparable to a 10-word lexicon.
  • Computational Efficiency: The system demonstrated fast decoding times, with text line processing taking less than a second for lexicons up to 300 words, which is considered quite fast.

Limitations and Future Work

  • Numerical Field Extraction: The authors acknowledge that the current baseline-dependent feature extraction is not ideal for numerical information, leading to lower performance in this area.
  • Future Work:
    • Comparison with Full Recognition: The authors plan to compare their system's performance (both in accuracy and speed) against information extraction systems that rely on full document recognition.
  • Improving Decoding: They aim to enhance the HMM decoding process by incorporating discriminative classifiers, potentially neural networks.

Summary updated

Abstract

In this paper, we introduce an alpha-numerical se-quences extraction system (keywords, numerical fields or alpha-numerical sequences) in unconstrained hand-written documents. Contrary to most of the approaches presented in the literature, our system relies on a global handwriting line model describing two kinds of infor-mation: i) the relevant information and ii) the ir-relevant information represented by a shallow parsing model. The shallow parsing of isolated text lines allows quick information extraction in any document while re-jecting at the same time irrelevant information. Results on a public french incoming mails database show the efficiency of the approach. 1.
BibTeX
@misc{simon2015alpha,
  author = {Thomas Simon and Clément Chatelain and Laurent Heutte},
  title = {{Alpha-numerical sequences extraction in handwritten documents}},
  year = {2015},
}
Figure