Introduction
Modern Transformer models face a significant bottleneck: the quadratic computational cost of self-attention relative to sequence length. This "limited horizon" prevents models from effectively processing long-range dependencies, such as those found in books or long conversations. Memory-Augmented Transformers (MATs) address this by offloading information to external databases. However, determining an optimal "encoding policy"—deciding which specific information to store—remains an open challenge. This research proposes that AI can benefit from "linking hypotheses" derived from cognitive neuroscience, specifically using surprisal as a signal for memory encoding.
The Surprisal Hypothesis
The authors investigate whether surprisal—the negative log-likelihood of an event—serves as a cross-domain signal for memory. In humans, prediction errors (high surprisal) often trigger the brain to encode information into long-term memory. To test this in AI, the researchers compared GPT-2's surprisal scores against human performance in a "cloze task" (word prediction) based on a 7.5-minute spoken story ("Pieman").
Results and Key Insights
The study revealed a robust link between model-based surprisal and human memory. GPT-2 surprisal scores significantly predicted which words participants would remember, showing a strong inverse correlation with cloze performance.
Crucially, the research debunked a common AI assumption: that "attention weights" are a proxy for importance. The study found that attention weights from GPT-2 showed almost no correlation with human memory performance. This suggests that while attention is vital for immediate processing, it does not necessarily indicate what is worth storing for the long term.
Implications for AI Architecture
These findings provide empirical support for using surprisal as a biologically-inspired encoding policy for MATs. If a model finds a word or concept difficult to predict, that high-surprisal moment acts as a "mental bookmark," signaling that the information should be saved to external memory. This approach moves beyond ad-hoc engineering toward a formal taxonomy of memory-augmented AI that mirrors human psychological processes of encoding and retrieval.
Limitations and Future Work
While promising, the study is limited by the relatively short length of the narrative used (approx. 1,000 tokens). Future research should explore whether the surprisal-memory link persists over much longer durations and more complex tasks, such as free-recall summarization, to further validate these linking hypotheses.
Summary updated