Comparison of Human and Machine Word Recognition

Markus Schenkel, Cyril Latimer, Marwan A. JabriNeurIPS 1997

This paper, hailing from the University of Sydney, provides a fascinating head-to-head comparison between human readers and a state-of-the-art (for the late 90s) Optical Character Recognition (OCR) system. By testing both on "tortured" text—documents degraded by repeated photocopying and faxing—the authors reveal where machine vision excels and where the human brain still holds the crown.

The Challenge of "Real-World" Text

While OCR is largely a solved problem for clean, high-resolution scans (often boasting 99.5%+ accuracy), it quickly falls apart when faced with real-world messiness. Low resolution (e.g., 200 DPI), noise, blur, and skew cause characters to either merge together or break into fragments.

The authors wanted to establish a benchmark: exactly how far is machine reading from the "gold standard" of human reading? Specifically, they investigated how much humans and machines rely on lexical context (knowing a word exists in a dictionary) versus raw character recognition when the visual signal is exceptionally poor.

Methodology: Man vs. Machine

The study employed a two-pronged approach to compare reading capabilities:

  1. The Human Experiment: 36 participants were shown 144 character strings each. These included real words of varying frequency and non-words. Some non-words were "legal" (pronounceable, like slunk) and others were "illegal" (random strings, like ptvca).
  2. The Machine Model: The authors developed a hybrid reading system using:
  3. Space Displacement Neural Network (SDNN): A multi-layer feed-forward network that estimates the probability of characters within a sliding window.
  4. Hidden Markov Model (HMM): This handled the "duration" of characters, helping the system decide where one character ends and the next begins (segmentation).
  5. Dictionary Model: A two-step process where the system first used "edit-distance" (how many letters to change to make a real word) to find candidates, then used the HMM to pick the most likely match.

Datasets and Degradation

The stimuli consisted of 36 data sets generated from the 44,000 most frequent words in the Sydney Morning Herald. Physical degradation was achieved by repeatedly photocopying and faxing documents before scanning them at 200 DPI. The machine model was trained on the SEDAL database, containing 180,000 characters from similarly low-resolution, degraded documents.

Key Insights

  • The "Closing of the Gap": As text becomes more unreadable, the performance gap between recognizing a real word and a non-word closes. In total degradation, everything approaches a 100% error rate. Both humans and machines showed a similar trajectory in how they lose accuracy as noise increases.
  • Segmentation Sensitivity: Machines are much more "grateful" for segmentation hints than humans. When the researchers provided markers showing where characters were separated, the machine's performance jumped significantly. Humans, conversely, found these markers slightly distracting and took longer to process them.
  • Contextual Parity: One of the most surprising findings was that the computer model used word context (the dictionary) almost exactly as efficiently as humans did for low-to-medium degradation.
  • Frequency vs. Logic: Humans are faster and more accurate with common words (the "frequency effect"). The machine, treating every dictionary entry as equally likely, showed no such bias.

Key Results

Accuracy: At low and medium degradation levels, the machine matched human performance on segmented data. However, on non-segmented data (the harder, more realistic task), the machine was about 7% worse than humans. At the highest level of degradation, humans significantly outperformed the machine, proving that human "wetware" is still superior at extracting signal from extreme noise.

Comparison of human and machine error rates across degradation levels for segmented and non-segmented text.

Speed: Non-words took longer for both humans and machines to process. For the machine, this was because non-words generated longer "candidate lists" from the dictionary, requiring more computation to reject them. Human reading times similarly spiked when faced with non-words and higher degradation.

Human reading times showing increased processing time for non-words and higher degradation.

Limitations and Future Work

The authors acknowledge that their model is a functional one, not a biological one. It lacks "higher-level" human context like syntax (sentence structure), semantics (meaning), and repetition priming (the fact that seeing a word once makes it easier to see again).

Future work suggested by the paper involves incorporating n-grams (Markov chains of letter sequences) to help the machine recognize "legal" non-words more like a human does, and moving beyond isolated word recognition into full sentence processing.

Summary updated

BibTeX
@inproceedings{schenkel1997comparison,
  author = {Markus Schenkel and Cyril Latimer and Marwan A. Jabri},
  title = {{Comparison of Human and Machine Word Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure