Confidence Prediction for Lexicon-Free OCR

Noam Mor, Lior Wolf

TL;DR

This paper introduces two novel methods for calculating confidence scores in lexicon-free Optical Character Recognition (OCR) systems. By moving beyond simple probability outputs, the researchers developed techniques—a probability ratio and a dedicated error-prediction sub-network—that allow OCR systems to "know when they don't know," significantly reducing misreads in real-world applications like processing dollar amounts or license plates.

What Problem Does This Paper Address?

In industrial OCR applications, accuracy isn't the only metric that matters; reliability is king. Most state-of-the-art OCR systems are "lexicon-based," meaning they rely on a dictionary to correct guesses (e.g., changing "he1lo" to "hello"). These systems have built-in confidence measures based on how well a word fits the dictionary.

However, many real-world tasks are "lexicon-free." Think of Social Security numbers, license plates, or handwritten dollar amounts—there is no dictionary for these. Without a lexicon, standard deep-learning OCR models (often using Connectionist Temporal Classification, or CTC) struggle to provide a meaningful confidence score. In a production pipeline, if a system can't flag its own mistakes, it can't decide when to hand a difficult image over to a human typist, leading to costly errors.

Methods

The authors build upon a state-of-the-art architecture (STAR-net) consisting of a CNN for feature extraction, a bidirectional LSTM for sequence modeling, and a CTC transcription layer. They propose two distinct ways to extract confidence:

  1. Probability Ratio (The "Runner-Up" Method): Instead of just looking at the probability of the top result ($p_1$), the system looks at the second-best guess ($p_2$). The confidence is calculated as $1 - (p_2 / p_1)$. If the top guess is significantly more likely than the runner-up, confidence is high. If they are close, the image is likely ambiguous, and confidence is low.
  2. Error Prediction Branch (The "Internal Critic"): This is a secondary neural network attached to the main OCR model. It takes intermediate activations from the first LSTM layer and is trained specifically to predict a binary outcome: "Will the OCR model get this right, or should it be rejected?" This branch is trained to minimize cross-entropy loss against the actual success/failure of the main model.

Datasets

The researchers evaluated their methods on a mix of standard benchmarks and a massive real-world dataset:

  • Scene Text: SVT (Street View Text) and ICDAR 2013.
  • Handwriting: IAM (English) and RIMES (French).
  • Proprietary: Orbograph’s Dollar Amount Dataset. This contains 1 million labeled images of scanned dollar amounts from printed documents. Crucially, it includes "reject" samples (blurry images, bad crops, or unrelated text) that mimic real-world messy data.

Key Insights and Ideas

  • Raw Probability is Deceptive: A common baseline is to use the raw CTC probability ($p$) as confidence. However, the authors found this performs poorly because $p$ naturally tends to decrease as the image (and the resulting character string) gets longer, regardless of actual accuracy.
  • The Value of Rejection: The authors argue that a system with 90% accuracy that can guarantee 99% accuracy on its most confident 50% of samples is more valuable to industry than a 98% accurate system that provides no confidence score.
  • Overfitting the Critic: On small datasets, the error-prediction branch can fail because the main OCR model might overfit and get 100% training accuracy. If the OCR model is "never wrong" during training, the critic branch never learns what an error looks like. To solve this, the authors suggest dedicating a specific subset of data solely to training the error-prediction branch.

Key Results

The proposed methods significantly outperformed standard baselines (raw CTC probability and length-normalized probability) across all datasets.

  • Area Under Curve (AUC): On the SVT dataset, the "Probability Ratio" achieved an AUC of 0.937, compared to just 0.484 for raw CTC probability.
  • Industrial Impact: On the Orbograph dataset, the Error Prediction model was able to reduce the error rate by as much as 99.7% while rejecting only 1% of the valid samples.
  • Qualitative Findings: When looking at the bottom 1st percentile of confidence scores, the model successfully flagged images that were handwritten (in a printed-text model), blurry, or contained "noise" like box edges from bad cropping.

Limitations and Future Work

  • Data Hunger: The "Error Prediction Branch" requires a significant amount of data to be effective. On smaller datasets like IAM or RIMES, the simpler "Probability Ratio" method actually performed better.
  • Architecture Specifics: While the authors claim the error-prediction sub-network is architecture-agnostic, it was only tested on a CNN+LSTM+CTC pipeline. Future work could explore how these confidence measures translate to newer Transformer-based OCR architectures.
  • Human-in-the-loop Optimization: The paper focuses on the prediction of the score but doesn't deeply explore the optimal thresholding strategies for balancing human labor costs versus error costs in a live environment.

Summary updated

Abstract

Having a reliable accuracy score is crucial for real world applications of OCR, since such systems are judged by the number of false readings. Lexicon-based OCR systems, which deal with what is essentially a multi-class classification problem, often employ methods explicitly taking into account the lexicon, in order to improve accuracy. However, in lexicon-free scenarios, filtering errors requires an explicit confidence calculation. In this work we show two explicit confidence measurement techniques, and show that they are able to achieve a significant reduction in misreads on both standard benchmarks and a proprietary dataset.
BibTeX
@inproceedings{mor2018confidence,
  author = {Noam Mor and Lior Wolf},
  title = {{Confidence Prediction for Lexicon-Free OCR}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=r8Usnai02pg},
}

Loading related content...

Figure