Abstract
Traditional handwriting recognition systems rely on computationally expensive lexicon-driven decoding, which struggles with large vocabularies. This paper introduces a novel "Lexicon Verification" paradigm using a cohort of LSTM models harvested from a single training run. By replacing exhaustive search with a fast verification cascade, the proposed method achieves state-of-the-art performance on benchmark datasets while scaling to vocabularies exceeding 3 million words.
Introduction
Handwriting recognition typically involves two distinct stages: an optical model that predicts character sequences and a linguistic model (lexicon) that constrains these predictions to valid words. Historically, this second stage has relied on lexicon-driven decoding, such as Viterbi or beam search, to find the most likely word in a dictionary.
As shown in Figure 1, the traditional approach forces the system to search through a lexicon to decode the character recognition output. This creates a significant computational bottleneck as the vocabulary size increases. Most current systems are limited to roughly 200,000 words, making them ill-suited for real-world applications involving diverse names, addresses, or technical terminology.
Proposed Methodology: Lexicon Verification and Model Cohorts
The authors propose a Lexicon Verification strategy within a Cascade Architecture to bypass the search bottleneck.
Lexicon Verification
Instead of searching for the best match within a lexicon during decoding, the system performs "Best Path Decoding" to predict characters freely. It then performs a simple membership check: does the predicted string exist in the lexicon? If yes, the word is accepted. If not, it is passed to the next model in a cascade. This verification is computationally trivial (O(1) with a hash set) regardless of lexicon size.
The LSTM Cohort
To build an effective cascade without the prohibitive cost of training hundreds of separate networks, the researchers utilize a "training trick." They found that models sampled from different epochs near the end of a single training run occupy different local minima and are thus complementary. This allows for the creation of a "cohort" of hundreds of diverse models from a single training session.
Minimum Number of Decision Agreement (MNDA)
To mitigate "False Acceptances"—where a model incorrectly predicts a string that happens to be a valid word—the system requires a specific number of models in the cascade to agree on the same word before acceptance. This thresholding is particularly critical for short words, which are statistically more likely to result in accidental lexicon matches.
Experimental Results
The method was evaluated on the RIMES (French) and IAM (English) datasets, using custom "gigantic" lexicons containing over 3.2 million and 2.4 million words, respectively.
- Accuracy: The system achieved a Word Error Rate (WER) of 3.00% on RIMES and 5.04% on IAM, both representing significant improvements over previous state-of-the-art benchmarks.
- Scalability: Performance remained robust even as the lexicon expanded to millions of words. On RIMES, accuracy only decreased from ~95% to ~90% when moving from a 5,000-word to a 3-million-word lexicon.
- Efficiency: By pruning the cohort to the 118 most effective networks, the system achieved a mean processing time of 197ms per word while maintaining high accuracy.
Conclusion
This research demonstrates that lexicon-driven decoding is not a prerequisite for high-accuracy handwriting recognition. By leveraging model diversity from a single training run and a fast verification paradigm, the proposed system effectively solves the large-vocabulary problem, offering a scalable and efficient alternative for real-world OCR applications. Future work will explore extending this logic to full-line text recognition and automating the MNDA thresholding process.
Summary updated