Abstract
Offline handwritten text line recognition is a hard task that requires both an efficient optical character recognizer and language model. Handwriting recognition state of the art methods are based on Long Short Term Memory (LSTM) recurrent neural networks (RNN) coupled with the use of linguistic knowledge. Most of the proposed approaches in the literature focus on improving one of the two components and use constraint, dedicated to a database lexicon. However, state of the art performance is achieved by combining multiple optical models, and possibly multiple language models with the Recognizer Output Voting Error Reduction (ROVER) framework. Though handwritten line recognition with ROVER has been implemented by combining only few recognizers because training multiple complete recognizers is hard. In this paper we propose a Lexicon Verified ROVER: LV-ROVER, that has a reduce complexity compare to the original one and that can combine hundreds of recognizers without language models. We achieve state of the art for handwritten line text on the RIMES dataset.
Introduction
Handwriting recognition typically relies on two pillars: an optical model to interpret characters and a language model to provide context. While Long Short-Term Memory (LSTM) networks have significantly advanced optical modeling, peak accuracy often requires combining multiple systems. The standard framework for this is ROVER (Recognizer Output Voting Error Reduction). However, ROVER faces two major scalability bottlenecks: the NP-complete complexity of aligning multiple text sequences using Dynamic Programming, and the immense resource cost of training dozens of complete (optical + language) models.
To address these challenges, we introduce LV-ROVER, a scalable evolution designed to combine hundreds of "raw" optical models (LSTMs without integrated language models) efficiently.
Methodology
The LV-ROVER Framework
LV-ROVER simplifies the combination process into two distinct modules:
- Alignment Module: Instead of computationally expensive Dynamic Programming, this module employs a fast heuristic. It performs a majority vote on the number of spaces in input strings to estimate word counts, then aligns words accordingly to create a word lattice. This reduces complexity from high-order polynomial to $O(N \times l)$, where $N$ is the number of models.
- Voting Module (Lexicon Verification): The word lattice is converted into a graph where the system searches for the optimal path based on word frequency. Crucially, it prioritizes "lexicon-verified" words found in a dictionary, serving as a lightweight and efficient substitute for traditional N-gram language models.
Cohort Generation
To generate the hundreds of models required for effective voting without prohibitive training costs, we exploit the nature of neural network loss surfaces. By saving model checkpoints at different epochs during a single training run—using specific parameters like high momentum—we extracted a cohort of 454 complementary models from just four training procedures.
Experimental Results
We evaluated LV-ROVER on the RIMES Dataset, a benchmark for handwritten French text, using various lexicons ranging from task-specific (8.6K words) to a "Gigantic Lexicon" (3.3M words).
LV-ROVER achieved state-of-the-art performance with a Character Error Rate (CER) of 2.53% and a Word Error Rate (WER) of 7.84%. Notably, the system proved robust to lexicon size; using a 3.3M word dictionary only slightly impacted performance compared to the small, task-specific one, suggesting high generalizability for open-vocabulary tasks.
As shown in the results, the Lexicon Verification step is critical. When using the same 454 models, standard ROVER lagged significantly behind (CER 4.73%), proving that simple voting is insufficient without the linguistic guidance provided by the LV-ROVER framework.
Conclusion
LV-ROVER demonstrates that aggregating the "wisdom of the crowd" from hundreds of raw optical models can outperform complex, high-order language models. The space-counting heuristic provides the necessary speed for large-scale ensembles, while lexicon verification ensures linguistic accuracy. Future work will explore extending this framework to multilingual recognition and multi-language document processing.
Summary updated