Cohort of LSTM and lexicon verification for handwriting recognition with gigantic lexicon

Bruno Stuner, Clément Chatelain, Thierry Paquet

TL;DR

This paper introduces a novel handwriting recognition paradigm that replaces traditional lexicon-driven decoding with a lexicon verification process within a cascade of LSTM networks. This approach achieves state-of-the-art performance, even with extremely large lexicons of millions of words, by efficiently combining complementary classifiers trained in a single session.

What Problem Does This Paper Address?

Traditional handwriting recognition systems rely on lexicon-driven decoding, which involves matching recognized character sequences against a predefined dictionary. While effective, these systems are limited by the size of their lexicons. Even the best systems, with hundreds of thousands of words, struggle to cover the vastness of real-world language, leading to "out-of-vocabulary" words and reduced overall performance. This limitation is particularly problematic for applications requiring broad language coverage, such as recognizing named entities or handling multilingual documents. The paper aims to overcome this by proposing a method that can effectively handle gigantic lexicons without sacrificing performance or introducing excessive computational cost.

Methods

The core of the proposed method is a cascade framework of Long Short-Term Memory (LSTM) recurrent neural networks. Instead of a single, large lexicon-driven decoder, the system uses a series of specialized LSTM "classifiers." The key innovations are:

  • Lexicon Verification Operator: This operator acts as a decision stage after each classifier. It accepts a word hypothesis if it exists in the lexicon and rejects it otherwise. This is a much faster process than traditional lexicon-driven decoding.
  • Cohort of Complementary Networks: The paper proposes generating hundreds of complementary LSTM networks from a single training session. This "cohort" is created by exploiting the properties of deep neural network training, where exploring different local minima can yield networks with similar performance but distinct internal representations. This "cohort" approach makes the learning process efficient.
  • Cascade Architecture: The complementary networks are arranged in a cascade. Each network processes the "rejects" from the previous one. The cascade is ordered by increasing error rate (decreasing reliability), and a Minimum Number of Decision Agreement (MNDA) is used to control the decision stage, further reducing false acceptances.

Datasets

The experiments were conducted on the following datasets:

  • Rimes: An isolated words dataset.
  • IAM: An isolated words dataset.

The paper also introduces and uses two extremely large lexicons for evaluation:

  • A French lexicon of 3,276,994 words, created by combining Wikipedia, Wiktionnaire, Gutenberg dictionary, and the Rimes dataset.
  • An English lexicon of 2,439,432 words, created by combining the one billion word dataset and the IAM dataset.

Key Insights and Ideas

  • Lexicon Verification as a Powerful Rejection Mechanism: The lexicon verification operator, when combined with a Minimum Number of Decision Agreement (MNDA), proves to be a highly effective and reliable way to reject incorrect hypotheses, significantly reducing the Word Error Rate (WER).
  • Complementarity from a Single Training: The paper demonstrates that a large number of complementary classifiers can be efficiently generated from a single training run by exploiting the exploration of local minima in deep neural networks. This "cohort" approach is key to the method's efficiency.
  • Low Sensitivity to Lexicon Size: The proposed cascade system exhibits remarkably low sensitivity to lexicon size, performing well even with lexicons containing millions of words. This is a significant advancement over traditional methods.
  • Cascade Efficiency: The cascade architecture, combined with the lexicon verification operator, allows for early rejection of hypotheses, significantly speeding up the overall processing time, especially when dealing with large numbers of classifiers.

Key Results

  • State-of-the-Art Performance: The proposed method achieves new state-of-the-art performance on both the Rimes and IAM datasets.
  • Gigantic Lexicon Handling: On a lexicon of over 3 million words, the system achieves a Word Recognition Rate (WRR) of 90.14% and a Word Error Rate (WER) of 9.18%.
  • Significant WER Reduction: Adding the lexicon verification strategy to a lexicon-free BLSTM recognizer on the Rimes dataset dramatically reduced the WER by 93%, from 33.63% to 2.25%.
  • MNDA Effectiveness: Using MNDA (e.g., 2 or 3) drastically reduces the probability of false acceptance (PFA) to very low levels, even for short words.
  • Processing Time: Despite using hundreds or thousands of networks, the cascade architecture and lexicon verification allow for fast processing. For a system with 2100 networks, the mean processing time per word is 729 milliseconds, with 80% of words processed in less than 0.175s.

Limitations and Future Work

  • Memory Usage: While not a major issue in this work, using a very large number of networks could potentially lead to memory concerns, although the authors note that 2100 networks fit within 6GB.
  • Future Work:
    • Extend the approach to line-level recognition using frameworks like ROVER.
  • Explore applications in multilingual document processing and named entity recognition, leveraging the ability to handle large lexicons.
  • Investigate the genericity of the cohort generation process with alternative architectures like convolutional neural networks or adversarial neural networks.

Summary updated

Abstract

State-of-the-art methods for handwriting recognition are based on Long Short Term Memory (LSTM) recurrent neural networks (RNN), which now provides very impressive character recognition performance. The character recognition is generally coupled with a lexicon driven decoding process which integrates dictionaries. Unfortunately these dictionaries are limited to hundred of thousands words for the best systems, which prevent from having a good language coverage, and therefore limit the global recognition performance. In this article, we propose an alternative to the lexicon driven decoding process based on a lexicon verification process, coupled with an original cascade architecture. The cascade is made of a large number of complementary networks extracted from a single training (called cohort), making the learning process very light. The proposed method achieves new state-of-the art word recognition performance on the Rimes and IAM databases. Dealing with gigantic lexicon of 3 millions words, the methods also demonstrates interesting performance with a fast decision stage.
BibTeX
@article{stuner2020cohort,
  author = {Bruno Stuner and Clément Chatelain and Thierry Paquet},
  title = {{Cohort of LSTM and lexicon verification for handwriting recognition with gigantic lexicon}},
  journal = {Multimedia Tools and Applications},
  year = {2020},
  doi = {10.1007/s11042-020-09198-6},
}
Figure