Marten van Schijndel, Tal Linzen
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Abstract
This paper explores a fundamental gap between how humans read and how standard AI models process text: adaptation. While humans constantly update their expectations based on the style, genre, and structure of what they are currently reading, most neural language models (LMs) remain static after their initial training. The authors demonstrate that a simple "online" update mechanism allows an LSTM to mimic human-like adaptation, significantly improving its ability to predict human reading behavior and showing that neural networks can learn abstract syntactic patterns on the fly.
Introduction
In psycholinguistics, it is well-established that humans read faster when a word matches their expectations. Crucially, these expectations are dynamic. If you are reading a legal contract, you start expecting "legalese"; if you see several "garden path" sentences, you eventually stop being "tricked" by them. Standard neural LMs, however, are typically "frozen" after training, failing to account for this local adaptation. This research investigates whether a simple adaptation mechanism can improve the psycholinguistic plausibility of an LM and whether the model can adapt to abstract structural patterns (syntactic adaptation) rather than just specific words (lexical adaptation).
Methodology
The researchers used a standard LSTM (Long Short-Term Memory) language model previously trained on 90 million words of Wikipedia. Their "Simple Adaptation" approach involves performing a single gradient update on the model's weights after every sentence, based on the cross-entropy loss (surprisal) of the actual words encountered. To link the model to human behavior, they used surprisal—a measure of how unpredictable a word is given the preceding context—and correlated it with self-paced reading time data from the Natural Stories Corpus.
Key Insights: Syntactic vs. Lexical Adaptation
A critical finding of the study is that the model does not merely get better because it sees the same words twice. In experiments with controlled syntactic stimuli like dative alternations (e.g., "The boy threw the ball to the dog" vs. "The boy threw the dog the ball"), the model showed improved performance on new sentences that shared the structure but had entirely different vocabulary.
The researchers identified a "Goldilocks" learning rate for this process. While lexical adaptation is relatively robust, syntactic adaptation is more fragile. If the learning rate is too high, the model overfits to specific words and fails to generalize the underlying grammar. A moderate learning rate (around 2.0 in their tests) was found to be optimal for learning abstract structures.
Experimental Results
The adaptive model significantly outperformed the static version across several key metrics:
- Perplexity Boost: On the Natural Stories corpus, adaptation dropped the model's perplexity (a measure of uncertainty) from 141.49 down to 86.99.
- Human Reading Times: Adaptive surprisal was a much stronger predictor of human reading speed ($t = 13.42$) than the non-adaptive version. When both measures were used in a regression model, the static version lost its predictive power, suggesting the adaptive model captures the essential dynamics of human reading.
- Resilience to Forgetting: Despite frequent weight updates, the model did not suffer from "catastrophic forgetting." It retained its ability to process general language significantly better than a non-adapted model even after adapting to a new genre.
Conclusion and Future Work
This work demonstrates that simple online updates allow neural networks to capture the dynamic nature of human language processing. While the current model resets to its base state after every story, future work could explore more sophisticated ways to initialize adaptation based on "environmental factors" like speaker identity or specific genres, rather than starting from scratch for every new text. The distinction between short-term priming and long-term adaptation remains a fertile ground for further research.
Summary updated