Language models can predict how long people spend reading each word. The signal is surprisal, the negative log-probability a model assigns to the word given what came before. Better models used to give better predictions of reading times. Past a point, that trend reverses: larger, better-trained models predict human reading worse, because they track long-range regularities that human memory does not hold. The usual remedies are crude. One adds a fixed penalty on distant words to the model's attention. Another fine-tunes the whole model directly on reading times, which damages its language ability. This note proposes a third option. Keep the model frozen, and train a small module that decides, for each word, which earlier words the "reader" still has access to and how deeply the current word is processed. Low-rank adapters let the frozen model work sensibly on the thinned-out context.
In two sentences: human reading samples text sparsely and forgets selectively, so we model the reader as a learned, content-dependent rule for dropping context, placed on top of a frozen language model. If surprisal computed under that learned forgetting fits reading times better than both a fixed recency penalty and full-model alignment, while the base model stays intact, we will have a mechanistic account of the inverse-scaling gap rather than a patch for it.
Core Claim
The claim is that the gap between model surprisal and human reading cost comes mostly from which context is available, and that this availability depends on content (what the words are), not only on distance. A learned, discrete keep-or-drop decision per context token should therefore close more of the gap than any static decay. It should also transfer across reading corpora where static decay fails.
The decision is made cheaply by a gate, which is the sense of conditional computation used here: parameters and context are activated selectively per input. The gate is trained with the straight-through discrete selection machinery used for video frame selection. Low-rank adaptation (LoRA, which learns a small additive update $BA$ to frozen weight matrices) is the constraint that keeps the base model recoverable.
The proposal also makes a falsifiable prediction about scale. Under learned forgetting, larger models should stop being worse predictors of reading times than smaller ones. If they remain worse, the inverse-scaling effect is not about context access, and the hypothesis fails.
Where This Sits
Three findings motivate the work. Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens locates where the fit to reading times peaks during training. The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage rules out memorization of the test texts as the cause. Language Models Are Better Than Humans at Next-token Prediction and On the Role of Context in Reading Time Prediction attribute the divergence to models exploiting context that humans do not retain, and to models separating contextual information from raw word frequency. All of these point to context access as the lever. None manipulates context access selectively.
Two methods constrain the design by showing what goes wrong. Locally Biased Transformers Better Align with Human Reading Times adds a fixed exponential distance penalty to GPT-2 attention. It improves fit on some corpora and worsens it on Natural Stories, and the authors themselves call static decay unrealistic. We replace the fixed penalty with a learned, content-conditioned drop decision. Reverse-Engineering the Reader fine-tunes all of GPT-2's weights to fit reading times. It fits a linear regression from surprisal to reading time in closed form at each step and adds a KL penalty that keeps the model close to the original. It gains substantial fit but raises perplexity and surprisal variance. We reuse its objective, but we train only a gate and low-rank adapters, so switching the module off returns the original model exactly.
Large Language Models Are Human-Like Internally shows that reading the model's prediction out of intermediate layers, by projecting their hidden states to word probabilities, fits human data better than the final layer in about 80% of cases. Early layers match early eye fixations and deeper layers match later integration measures. That analysis is diagnostic and picks one fixed layer. We let depth vary per token, and we treat "best fixed layer" as a baseline the proposal must beat.
Two methods are reused. Flexible Frame Selection for Efficient Video Reasoning learns which video frames to keep, trained end to end with a Gumbel straight-through estimator and placeholder tokens that mark discarded slots. We apply the same estimator to discard text tokens from a reader's memory. The analogy is literal: a reader's fixations are a sparse temporal sampling of the text, with irreversible loss of what was skipped or forgotten. Mixture of LoRA Experts supplies the layer-wise gating over low-rank modules that we use for depth routing. We give it a balancing loss for the same reason that paper does, to keep the gate from always choosing one option. Xu reports that low-rank updates preserve calibration better than full fine-tuning, which motivates the expectation that LoRA limits damage. We test that expectation rather than assume it.
Research Program
The base is a frozen causal language model; we use the GPT-2 family to match prior work, plus a larger model for the scaling test. The reader module has two parts.
The first is a retention gate. When token $j$ is read, a small network takes its early-layer hidden state and outputs a keep probability $p_j$. A hard keep-or-drop mask $m_j \in \{0,1\}$ is sampled from it with the Gumbel straight-through estimator. The gate never sees future words. When the model later computes surprisal for word $t$, attention to token $j$ is allowed only if the token is recent or was kept:
$$\text{visible}(t, j) = \mathbb{1}[\,t - j \le w\,] \;\lor\; m_j,$$
where $w$ is a short window of always-visible recent words. This is state-dependent forgetting, determined by the content of the forgotten word rather than its distance. A sparsity penalty on the mean of $p_j$ sets how much is forgotten. The penalty strength is tuned so that the fraction of tokens kept matches the comparisons described under Baselines.
The second is a depth gate. A per-token mixture, trained with a MoLE-style balancing loss, chooses among $K$ exits at 20, 40, 60, 80, and 100% depth. Each exit maps that layer's hidden state to word probabilities through a tuned-lens readout, a small learned linear map. Low-rank adapters on the attention and feedforward layers, of rank 8 to 16, let the frozen network adjust to contexts with holes in them.
Training maximizes the reading-time regression likelihood with closed-form coefficients, following Reverse-Engineering the Reader. Reading time $\text{RT}_t$ is regressed on the reader's surprisal $S_t$, the previous word's surprisal, word length, and word frequency. The regression's log-likelihood gain over a model without surprisal is $\Delta llh$. The loss adds a KL term toward the base model on intact context, which keeps the reader-mode model a competent language model. With the adapter and gates off, the original model is recovered exactly. The real cost is therefore measured in reader mode: perplexity on held-out text and surprisal variance, the property Reverse-Engineering the Reader found inflated. Surprisal variance also bears on the uniform-information-density principle, the idea that language spreads information evenly across words.
Analysis then asks what the gate learned. We check whether keep probability rises with word frequency, part of speech, and syntactic heads such as verbs and subjects of open dependencies. On eye-tracking corpora, we check whether low-retention words line up with words human readers skip, which can be derived from Provo's fixation records. We also check whether the chosen depth tracks the early-versus-late split reported in the internal-layers paper.
Baselines
The naive reference is the frozen base model's final-layer surprisal, entered into the same regression with the same spillover and control predictors. Every $\Delta$AIC and $\Delta llh$ in this note is measured relative to this point.
The strongest fixed-memory competitor is Locally Biased Transformers ( Locally Biased Transformers Better Align with Human Reading Times). That paper reports $\Delta$AIC relative to GPT-2 of −21.55 on MECO, −13.72 on Provo, −12.55 on Brown, and +4.31 on Natural Stories, where the method is worse than plain GPT-2; it reports parity on UCLET and UCLSPR. We reimplement it on the same base model. Its decay rate and blending factor are tuned by grid search on the same development folds, with the same number of trials as our gate's hyperparameters. Beating it means a lower $\Delta$AIC on Provo, MECO, and Brown, and a negative $\Delta$AIC on Natural Stories, the corpus where static decay fails. We also run a stronger version of this baseline in which the decay rate is learned by gradient descent, so the comparison isolates content-dependence rather than learning versus hand-tuning.
The strongest alignment competitor is Reverse-Engineering the Reader ( Reverse-Engineering the Reader), full fine-tuning with the regression and KL objective. It reports a ZuCo-to-ZuCo $\Delta llh$ increase of over 81% for GPT2-Large. No reported number was found on Lacuna for its Dundee or Provo gains. We rerun it with the same objective, data splits, and optimizer budget, and we also run it as LoRA-only, with no gates. Matching its $\Delta llh$ within its seed variance while keeping reader-mode perplexity and surprisal variance lower would count as winning. Exceeding its $\Delta llh$ on cross-corpus transfer, training on one corpus and testing on another, would be the stronger result, because full fine-tuning is the method most likely to overfit a single corpus.
The best fixed intermediate layer, read through Logit Lens or Tuned Lens ( Large Language Models are Human-Like Internally), is chosen per corpus on development data. No reported number was found on Lacuna for per-corpus $\Delta$AIC under this selection.
Ablations isolate the key ingredient. The first is a rate-matched random retention mask: the same fraction of tokens dropped, chosen at random. If it does as well as the learned gate, content-dependence is not doing the work. The other ablations remove one component at a time: gate without LoRA, retention gate alone, and depth gate alone. The final baseline is an early training checkpoint near two billion tokens, the point of best fit reported by Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens, when such checkpoints exist for the model family used.
Why not just…?
Why not just read surprisal from the best intermediate layer? This is the most serious threat. It needs no training and already beats the final layer in most cases. If the depth gate alone matches the full module and the fixed-layer baseline matches the depth gate, the retention mechanism adds nothing, and we will report that. The proposal's distinctive bet is the retention gate. The decisive comparison is retention gate plus best fixed layer against best fixed layer alone, and that comparison will settle this question.
Why not just tune or learn the decay rate of a locally biased transformer? A per-corpus decay can be fitted, but the reported Natural Stories result (+4.31 $\Delta$AIC) suggests distance alone is the wrong variable for texts with rare syntactic structures. The learned-decay version of this baseline gives distance-based forgetting its best chance. If it closes the Natural Stories gap as well as content-dependent retention does, our core claim is wrong.
Why not just use a smaller or earlier-checkpoint model? Weaker models fit reading times better, per the two-billion-token result, and are cheap. They do not explain why, and they cannot be used together with a capable model. We include them as baselines. The scaling experiment below tests whether learned forgetting makes model size irrelevant to fit, which the small-model alternative cannot show.
Why not just apply the Reverse-Engineering objective with LoRA, without gates? This ablation may recover much of the gain, since LoRA alone may limit damage to the model. If it matches the gated module on $\Delta llh$ and on reader-mode perplexity, the gates are unnecessary. The gated module must then win on cross-corpus transfer or on interpretability: gate decisions that line up with human skipping.
Proposed Experiments
The primary evaluation trains on each of Provo, Dundee, and ZuCo and tests within each corpus and across all reading corpora in the file: MECO, Brown, Natural Stories, UCLET, and UCLSPR. Metrics are $\Delta$AIC and $\Delta llh$ with 10-fold cross-validation over texts, reader-mode perplexity on held-out text, and surprisal variance.
There are three decisive checks. First, the learned gate must beat the rate-matched random mask and learned static decay on Natural Stories, where static decay fails. Second, the learned retention probabilities should be lower for words humans skip in Provo; a correlation near zero would mean the gate fits reading times for reasons unrelated to human sampling. Third, applying the identical module to GPT-2 small, medium, and large, plus one larger model, should flatten or reverse the inverse-scaling slope of $\Delta llh$ against model size. If larger models remain worse under learned forgetting, context access does not explain the gap.
If these checks pass, the same retention gate is a natural candidate for query-free, memory-limited frame dropping in long-video models. EgoSchema, for example, shows accuracy declining when more frames are added. We leave that transfer to follow-up work, because the file offers no human-alignment measure for video.
References
- Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
- The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage
- Language Models Are Better Than Humans at Next-token Prediction
- On the Role of Context in Reading Time Prediction
- Locally Biased Transformers Better Align with Human Reading Times
- Reverse-Engineering the Reader
- Large Language Models are Human-Like Internally
- Flexible Frame Selection for Efficient Video Reasoning
- Mixture of LoRA Experts
- On the Calibration of Large Language Models and Alignment
- Locally Biased Transformers Better Align with Human Reading Times
- Reverse-Engineering the Reader
- Large Language Models Are Human-Like Internally
- Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens