TL;DR
This paper introduces two novel methods for calculating confidence scores in lexicon-free Optical Character Recognition (OCR) systems. By moving beyond simple probability outputs, the researchers developed techniques—a probability ratio and a dedicated error-prediction sub-network—that allow OCR systems to "know when they don't know," significantly reducing misreads in real-world applications like processing dollar amounts or license plates.
What Problem Does This Paper Address?
In industrial OCR applications, accuracy isn't the only metric that matters; reliability is king. Most state-of-the-art OCR systems are "lexicon-based," meaning they rely on a dictionary to correct guesses (e.g., changing "he1lo" to "hello"). These systems have built-in confidence measures based on how well a word fits the dictionary.
However, many real-world tasks are "lexicon-free." Think of Social Security numbers, license plates, or handwritten dollar amounts—there is no dictionary for these. Without a lexicon, standard deep-learning OCR models (often using Connectionist Temporal Classification, or CTC) struggle to provide a meaningful confidence score. In a production pipeline, if a system can't flag its own mistakes, it can't decide when to hand a difficult image over to a human typist, leading to costly errors.
Methods
The authors build upon a state-of-the-art architecture (STAR-net) consisting of a CNN for feature extraction, a bidirectional LSTM for sequence modeling, and a CTC transcription layer. They propose two distinct ways to extract confidence:
- Probability Ratio (The "Runner-Up" Method): Instead of just looking at the probability of the top result ($p_1$), the system looks at the second-best guess ($p_2$). The confidence is calculated as $1 - (p_2 / p_1)$. If the top guess is significantly more likely than the runner-up, confidence is high. If they are close, the image is likely ambiguous, and confidence is low.
- Error Prediction Branch (The "Internal Critic"): This is a secondary neural network attached to the main OCR model. It takes intermediate activations from the first LSTM layer and is trained specifically to predict a binary outcome: "Will the OCR model get this right, or should it be rejected?" This branch is trained to minimize cross-entropy loss against the actual success/failure of the main model.
Datasets
The researchers evaluated their methods on a mix of standard benchmarks and a massive real-world dataset:
- Scene Text: SVT (Street View Text) and ICDAR 2013.
- Handwriting: IAM (English) and RIMES (French).
- Proprietary: Orbograph’s Dollar Amount Dataset. This contains 1 million labeled images of scanned dollar amounts from printed documents. Crucially, it includes "reject" samples (blurry images, bad crops, or unrelated text) that mimic real-world messy data.
Key Insights and Ideas
- Raw Probability is Deceptive: A common baseline is to use the raw CTC probability ($p$) as confidence. However, the authors found this performs poorly because $p$ naturally tends to decrease as the image (and the resulting character string) gets longer, regardless of actual accuracy.
- The Value of Rejection: The authors argue that a system with 90% accuracy that can guarantee 99% accuracy on its most confident 50% of samples is more valuable to industry than a 98% accurate system that provides no confidence score.
- Overfitting the Critic: On small datasets, the error-prediction branch can fail because the main OCR model might overfit and get 100% training accuracy. If the OCR model is "never wrong" during training, the critic branch never learns what an error looks like. To solve this, the authors suggest dedicating a specific subset of data solely to training the error-prediction branch.
Key Results
The proposed methods significantly outperformed standard baselines (raw CTC probability and length-normalized probability) across all datasets.
- Area Under Curve (AUC): On the SVT dataset, the "Probability Ratio" achieved an AUC of 0.937, compared to just 0.484 for raw CTC probability.
- Industrial Impact: On the Orbograph dataset, the Error Prediction model was able to reduce the error rate by as much as 99.7% while rejecting only 1% of the valid samples.
- Qualitative Findings: When looking at the bottom 1st percentile of confidence scores, the model successfully flagged images that were handwritten (in a printed-text model), blurry, or contained "noise" like box edges from bad cropping.
Limitations and Future Work
- Data Hunger: The "Error Prediction Branch" requires a significant amount of data to be effective. On smaller datasets like IAM or RIMES, the simpler "Probability Ratio" method actually performed better.
- Architecture Specifics: While the authors claim the error-prediction sub-network is architecture-agnostic, it was only tested on a CNN+LSTM+CTC pipeline. Future work could explore how these confidence measures translate to newer Transformer-based OCR architectures.
- Human-in-the-loop Optimization: The paper focuses on the prediction of the score but doesn't deeply explore the optimal thresholding strategies for balancing human labor costs versus error costs in a live environment.
Summary updated