Abstract
Language processing mechanism by humans is generally more robust than computers. The Cambridge University effect from the psycholinguistics literature has demonstrated such a robust word processing mechanism, where jumbled words (e.g., "Cmabrigde") are recognized with little cost. On the other hand, computational models for word recognition perform poorly on data with such noise. Inspired by these findings, we propose a word recognition model based on a semi-character level recurrent neural network (scRNN). In our experiments, we demonstrate that scRNN has significantly more robust performance in word spelling correction compared to existing spelling checkers and character-based convolutional neural networks. Furthermore, we demonstrate that the model is cognitively plausible by replicating a psycholinguistics experiment about human reading difficulty using our model.
Introduction: The "Cambridge University" Effect
While natural language processing (NLP) has advanced rapidly, computers remain surprisingly fragile when faced with "noisy" text. Humans can effortlessly read a sentence like "Aoccdrnig to a rscheearch...", but standard computational models—including sophisticated character-based neural networks—often break down when character order is slightly permuted. This phenomenon, known as the Cambridge University effect, highlights a significant gap between human psycholinguistic processing and machine word recognition.
The challenge addressed in this work is to develop a system for robust word recognition: creating a model that can handle jumbled letters, deletions, and insertions while remaining computationally efficient and cognitively plausible.
Methods: The scRNN Architecture
The proposed scRNN (semi-character Recurrent Neural Network) mimics human word recognition by focusing on word boundaries. Instead of processing a word as a strict sequence of characters, the scRNN represents each word $x_n$ as a concatenated vector of three parts:
- Beginning ($b$): A one-hot encoding of the first character.
- Internal ($i$): A "bag-of-characters" vector representing the counts of all characters between the first and last (ignoring their order).
- Ending ($e$): A one-hot encoding of the last character.
This input is fed into a standard Long Short-Term Memory (LSTM) network, which predicts the intended word via a softmax layer. By intentionally discarding the internal character order, the model becomes inherently invariant to internal jumbling—matching the robust processing mechanism observed in human readers.
Experimental Results
The scRNN was trained on the Penn Treebank (PTB) and evaluated against synthetic noise including jumbles, deletions, and insertions. The results demonstrate a significant leap in robustness over existing baselines:
- Jumbled Text: scRNN achieved 98.96% accuracy, while the state-of-the-art CharCNN (Character-based Convolutional Neural Network) managed only 17.17%. Even popular commercial spell-checkers hovered around 54-57%.
- Deletion/Insertion: scRNN maintained high performance (85.74% for deletions and 96.70% for insertions), outperforming other models by significant margins.
- Efficiency: The model is remarkably lightweight, using only 50 LSTM units (~2MB), and reaches 90% accuracy within just 15,000 iterations.
Cognitive Plausibility
A key contribution of this work is demonstrating that the scRNN is cognitively plausible. The authors replicated a psycholinguistic experiment (Rayner et al., 2006) measuring human reading difficulty across different jumbling conditions.
The scRNN’s "difficulty" (measured by error rate) followed the exact same pattern as human reading difficulty shown in the table above: Internal Jumble < Ending Jumble < Beginning Jumble. This reinforces the theory that the first and last letters of a word are the most critical anchors for human word recognition.
Conclusion and Future Work
The scRNN demonstrates that word recognition does not require massive parameter counts if the input representation is clever. While the model struggles with internal anagrams (e.g., "trial" vs. "trail") and capitalization, its inherent robustness makes it highly effective for normalizing social media text or post-processing OCR output. Future work will explore scaling this approach to open-vocabulary scenarios and morphologically rich languages.
Summary updated