Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network

Keisuke Sakaguchi, Kevin Duh, Matt Post, Benjamin Van Durme

Abstract

Human readers possess a remarkable ability to process jumbled text, provided the first and last letters of words remain in place—a phenomenon known as the "Cambridge University" effect. In contrast, modern Natural Language Processing (NLP) systems are often fragile when encountering even minor character-level noise. This paper introduces the semi-character Recurrent Neural Network (scRNN), a cognitively-inspired architecture that represents words as a combination of their first character, last character, and an unordered bag of internal characters. Our results demonstrate that the scRNN achieves nearly 99% accuracy in correcting jumbled words, significantly outperforming traditional spell checkers and character-level deep learning models.

Introduction

The robustness of human word recognition stands in stark contrast to the sensitivity of state-of-the-art NLP models. While humans can effortlessly read "Aoccdrnig to a rscheearch," a single typo can derail a character-level CNN or a standard LSTM. Traditional error correction methods often rely on edit distances, which do not account for the position-dependent nature of human perception. We propose a model that is "cognitively plausible," leveraging the psycholinguistic insight that the first and last letters of a word serve as critical anchors for recognition.

Methodology: The scRNN Architecture

The core innovation of this work is the semi-character Recurrent Neural Network (scRNN). Unlike standard models that process character sequences linearly, the scRNN employs a word-level representation that is invariant to internal permutations.

Each word is converted into a single input vector $x_n$ consisting of three distinct components:

  1. Beginning ($b$): A one-hot encoding of the first character.
  2. Internal ($i$): A "bag of characters" (count vector) representing all characters between the first and last, ignoring their specific order.
  3. Ending ($e$): A one-hot encoding of the last character.
The scRNN architecture processing jumbled input tokens.

This input is processed by a Long Short-Term Memory (LSTM) network, which predicts the correct word from a fixed vocabulary. The model is remarkably efficient; while larger configurations were tested, a version with just 50 hidden units (approximately 2MB) proved sufficient for high-accuracy recognition.

Experimental Setup

The model was trained and evaluated using the Penn Treebank (PTB) corpus. To test robustness, the researchers introduced three types of synthetic noise:

  • Jumble: Shuffling internal characters (e.g., Cambridge $\rightarrow$ Cmabrigde).
  • Delete: Removing one internal character.
  • Insert: Adding one extra internal character.

Results

The scRNN demonstrated exceptional performance, particularly on jumbled text. While a standard CharCNN achieved only 17.17% accuracy on jumbled words, the scRNN reached 98.96%. It also outperformed commercial spell checkers, which typically hovered around 55% accuracy.

Training accuracy of the scRNN across different hyperparameter settings.

As shown in the accuracy plots above, the model converges rapidly across various hyperparameter configurations ($\beta$ values). Beyond jumbling, the model showed surprising generalization to deletions (85.74%) and insertions (96.70%). Furthermore, the RNN structure allows the model to use sentence context to disambiguate anagrams (e.g., deciding between "mass" and "miss" based on surrounding words).

Cognitive Alignment and Discussion

A key finding is that the scRNN's performance degradation mirrors human difficulty. Psycholinguistic studies show that jumbling the beginning or ending letters of a word significantly increases human eye fixation time compared to internal jumbling. When the researchers modified the scRNN to jumble "anchor" positions, its accuracy dropped sharply, aligning the model's failure modes with human cognitive constraints.

Despite its success, the model has limitations. The "bag of characters" approach makes it susceptible to internal anagrams (e.g., "there" vs. "three") in ambiguous contexts. Additionally, the current implementation relies on a fixed vocabulary, limiting its ability to handle out-of-vocabulary (OOV) terms. Future work will focus on extending this architecture to handle more naturalistic noise, such as OCR errors and social media slang.

Summary updated

Abstract

Language processing mechanism by humans is generally more robust than computers. The Cmabrigde Uinervtisy (Cambridge University) effect from the psycholinguistics literature has demonstrated such a robust word processing mechanism, where jumbled words (e.g. Cmabrigde / Cambridge) are recognized with little cost. On the other hand, computational models for word recognition (e.g. spelling checkers) perform poorly on data with such noise. Inspired by the findings from the Cmabrigde Uinervtisy effect, we propose a word recognition model based on a semi-character level recurrent neural network (scRNN). In our experiments, we demonstrate that scRNN has significantly more robust performance in word spelling correction (i.e. word recognition) compared to existing spelling checkers and character-based convolutional neural network. Furthermore, we demonstrate that the model is cognitively plausible by replicating a psycholinguistics experiment about human reading difficulty using our model.
BibTeX
@inproceedings{sakaguchi2017robsut,
  author = {Keisuke Sakaguchi and Kevin Duh and Matt Post and Benjamin Van Durme},
  title = {{Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=sM5QgjDvl6V},
}
Figure