Neural Speed Reading Audited

Anders SøgaardEMNLP 2020 · OpenReview

Abstract: Anders Søgaard. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

Introduction

Between 2017 and 2020, the Natural Language Processing (NLP) community saw a surge in "Neural Speed Reading" (NSR) architectures. These models, typically based on Recurrent Neural Networks (RNNs) like LSTMs, were designed to mimic human reading behaviors—specifically the ability to "skip" irrelevant words or "jump" across sentences. The goal was to reduce the computational cost of processing long documents while maintaining high accuracy. However, a critical audit presented at EMNLP 2020 suggests that this subfield is built on a shaky scientific foundation and that its complex architectures are significantly outperformed by much simpler, older techniques.

The "Human-Inspired" Myth

The primary justification for NSR models is often their resemblance to human cognitive strategies, such as eye fixations and saccades. Proponents argue that by modeling how humans skim text, machines can achieve similar efficiencies.

Author Anders Søgaard identifies two major flaws in this reasoning. First, psycholinguistic evidence indicates that human "speed reading" is generally a flawed strategy; humans cannot skip significant portions of text without a substantial loss in comprehension. Thus, modeling speed reading is not modeling a successful cognitive strategy, but rather a compromised one. Second, the biological metaphor of "jumping" is inaccurate; humans are limited by the fovea and cannot extract high-quality information from large areas of text in a single glance.

Figure 1: A conceptual audit of the logic behind Neural Speed Reading models.

The Efficiency Paradox

Beyond the scientific validity of the human analogy, NSR models face an efficiency paradox. To implement "skipping" logic, these models often require complex mechanisms like Reinforcement Learning (e.g., A3C) or Gumbel-softmax tricks. These additions make the models harder to train and computationally heavier than the very baselines they seek to improve upon.

To test whether this complexity is necessary, the paper compares state-of-the-art NSR models against a "Simple MLP" baseline. This baseline is a basic Multi-Layer Perceptron with a single hidden layer that uses a "bag-of-n-grams" approach. Unlike sequential models, the MLP processes features all at once, making it inherently non-sequential and trivial to parallelize.

Benchmarking Simplicity

The study evaluated these models across three standard document classification benchmarks: IMDB (sentiment), RottenTomatoes (sentiment), and AG News (topic classification). Performance was measured by accuracy and FLOP reductions (FL-r), which quantifies the floating-point operations saved compared to a standard LSTM.

Figure 2: Performance comparison of Simple MLP against various Neural Speed Reading models and an LSTM baseline.

As shown in the table above, the results were a landslide victory for simplicity. The Simple MLP consistently outperformed the complex NSR models in both accuracy and speed across all tested datasets.

  • Superior Accuracy: The Simple MLP achieved a 7% average error reduction over the state-of-the-art Structural-Jump-LSTM.
  • Massive Speedup: On the IMDB dataset, the Simple MLP was over 1,101x faster than the LSTM baseline, whereas the leading NSR model (Hansen et al., 2019) only achieved a 6.3x speedup. On average, the MLP was roughly 136x faster than the NSR models it was compared against.

Conclusion

The audit concludes that the NLP community should reconsider the value of "human-inspired" speed reading models for document classification. The complexity required to mimic human skipping behavior often hinders the goal of efficiency. Instead, researchers should focus on computationally efficient NLP—prioritizing models that are fast due to mathematical simplicity and hardware compatibility. The success of the Simple MLP demonstrates that for many tasks, "fast NLP" is better achieved through architectural simplicity than through complex cognitive modeling.

Summary updated

Abstract

Anders Søgaard. Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.
BibTeX
@inproceedings{sgaard2020neural,
  author = {Anders Søgaard},
  title = {{Neural Speed Reading Audited}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=7J3nuY5v36-},
}
Figure