Abstract: Having a reliable accuracy score is crucial for real world applications of OCR, since such systems are judged by the number of false readings. Lexicon-based OCR systems, which deal with what is essentially a multi-class classification problem, often employ methods explicitly taking into account the lexicon, in order to improve accuracy. However, in lexicon-free scenarios, filtering errors requires an explicit confidence calculation. In this work we show two explicit confidence measurement techniques, and show that they are able to achieve a significant reduction in misreads on both standard benchmarks and a proprietary dataset.
Introduction
In industrial Optical Character Recognition (OCR) applications, reliability is often more critical than raw accuracy. While many state-of-the-art systems are "lexicon-based"—relying on dictionaries to correct guesses (e.g., correcting "he1lo" to "hello")—many real-world tasks are "lexicon-free." Tasks such as processing Social Security numbers, license plates, or handwritten dollar amounts lack a dictionary to guide error correction.
Without a lexicon, standard deep-learning OCR models, typically using Connectionist Temporal Classification (CTC), struggle to provide meaningful confidence scores. This paper introduces two novel methods for calculating confidence in these scenarios, allowing systems to identify when they are likely to fail and hand difficult cases over to human operators.
Methodology
The researchers utilize the STAR-net architecture, which consists of a CNN for feature extraction, a bidirectional LSTM for sequence modeling, and a CTC transcription layer. To derive confidence scores, they propose two distinct approaches:
- Probability Ratio: Instead of relying solely on the top result's probability ($p_1$), this method considers the runner-up guess ($p_2$). Confidence is calculated as $1 - (p_2 / p_1)$. A large gap between the top two guesses indicates high confidence, while a small gap suggests ambiguity.
- Error Prediction Branch: This is a secondary neural network integrated into the main OCR model. It consumes intermediate activations from the LSTM layer and is trained to predict whether the main model will succeed or fail on a given sample.
Key Insights and Results
A significant finding is that raw CTC probability is a deceptive confidence metric because it naturally decreases as character strings get longer, regardless of actual accuracy. The proposed methods significantly outperform standard baselines like raw CTC probability and length-normalized probability across various datasets, including SVT (Street View Text), IAM (handwriting), and a massive proprietary dataset of dollar amounts from Orbograph.
Performance on Standard Benchmarks
On the SVT dataset, the "Probability Ratio" method achieved an Area Under Curve (AUC) of 0.937, a dramatic improvement over the 0.484 achieved by raw CTC probability. This demonstrates the method's superior ability to distinguish between correct and incorrect reads.
Industrial Impact
On the Orbograph dataset, which includes messy real-world data like blurry images and bad crops, the Error Prediction model reduced the error rate by up to 99.7% while rejecting only 1% of valid samples. This level of reliability is vital for production pipelines where minimizing false readings is the priority.
Conclusion and Future Work
While the Error Prediction Branch is highly effective, it is data-hungry; on smaller datasets like IAM, the simpler Probability Ratio method often performs better. Future research could explore these confidence measures within newer Transformer-based architectures and investigate optimal thresholding strategies to balance human labor costs against error costs in live environments.
Summary updated