Language models often fail at chained spatial questions like "A is left of B, B is above C, … where is A relative to E?" A common reading is that they cannot compose spatial relations. This proposal tests a different explanation. Much of the failure may be a reading problem rather than a reasoning problem. The few sentences that form the reasoning chain are buried among distractor sentences, and often among worked examples in the prompt as well. Attention spread over all of that text drowns out the chain. If this is right, two well-known failures share one cause. The first is the multi-hop spatial gap. The second is in-context learning saturation, where adding more worked examples to a prompt eventually makes accuracy worse. Both should then respond to the same remedy: a filter that prunes the context coarse-to-fine before the model reads it, and that follows the chain one hop at a time so it does not throw away the "bridge" sentences linking the question's entities.
The work has two parts. The first is a diagnostic that separates reading failures from composition failures using oracle chain contexts. The second is a learned filter that expands from the query entities hop by hop and scores each sentence by how much it changes the model's answer. The headline prediction is concrete and falsifiable. After pruning, the number of demonstrations at which accuracy peaks should move to the right. That would mean the saturation point is not a fixed property of the model but depends on how much irrelevant text competes for attention.
Core Claim
The claim is that, for current language models, the textual multi-hop spatial gap is mostly a context-competition effect. Removing off-chain sentences, whether story distractors or excess demonstrations, recovers much of the lost accuracy. A pruner that grows its kept set hop by hop beats a flat sentence pruner because a flat pruner cannot tell an intermediate bridge sentence from noise. What would be new is three things. The first is the joint treatment of story distractors and prompt demonstrations as one competition for attention. The second is the measurement of how pruning shifts the saturation curve. The third is a hop-conditioned version of counterfactual sentence scoring, trained without supporting-fact labels.
The motivation comes from three findings. StepGame builds chained spatial stories with three injected noise types: irrelevant branches, disconnected chains, and redundant supporting paths. On this benchmark, relational networks fall to near chance at five hops. Advancing Spatial Reasoning in LLMs on StepGame corrects StepGame's template errors and shows that GPT-4 fails at direct multi-hop integration but succeeds when paired with Answer Set Programming, a logic solver. On the demonstration side, Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals finds that adding demonstrations raised GPT-3's MNLI accuracy while its attentiveness to the actual query dropped from 92.9% to 62.0%. RiOT attributes this kind of degradation to softmax attention spreading over a growing pool of tokens.
None of this work asks whether story distractors and demonstrations interact. They should interact if they compete for the same attention.
Research Program
Stage one: the diagnostic. For each StepGame item, we build four contexts. The first is the full noisy story. The second is the oracle chain alone, in story order. The third is the oracle chain shuffled. The fourth is the oracle chain plus each noise type added back separately. Each context is crossed with 0, 1, 2, 4, 8, 16 and 32 demonstrations. We use the corrected StepGame release from the Advancing paper so that template errors do not masquerade as reasoning failures.
From each accuracy-versus-demonstrations curve we read off $n^*$, the demonstration count at which accuracy peaks. We also measure attentiveness with the counterfactual test of Elazar et al. To do this, we swap the two query entities and check whether the answer flips to the opposite relation, as it should.
If oracle-chain accuracy at $k$ hops stays low, the gap is compositional and the method below is not warranted. In that case the paper reports that negative result. If oracle-chain accuracy is high and full-story accuracy is low, the gap is a reading gap, and the noise-type ablation tells us which distractors cause it.
Stage two: the hop-aware pruner. The pruner works in two passes, coarse then fine. The coarse pass is cheap. Starting from the question's two entities, it builds an entity graph over sentences and drops whole sentence blocks that are not reachable from either entity, which removes disconnected noise. The fine pass is iterative. It begins with the kept set $S_0$ containing the sentences that mention a query entity. At hop $t$ it scores each candidate sentence $s$ that shares an entity with $S_t$ by its marginal contribution:
$$m_t(s) = \log p_\theta(y \mid q, S_t \cup \{s\}) - \log p_\theta(y \mid q, S_t)$$
Here $q$ is the question, $y$ is the reader model's answer on the full context, and $p_\theta$ is a small encoder trained to predict that answer. The highest-scoring sentences are added to form $S_{t+1}$. The process stops when both query entities are connected or when no candidate's score is positive.
The scoring encoder reuses the training recipe of EnComp. EnComp is a ModernBERT compressor trained on counterfactual sentence redactions. It filters chunks and then prunes sentences in a single forward pass, with 45 ms latency and 1.86 GB of GPU memory. Our change is to condition each sentence's score on the current kept set rather than scoring every sentence independently. That conditioning is what lets a bridge sentence, worthless on its own, score highly once its neighbor has been kept.
Training labels come from redaction. We delete a sentence and record whether the reader's answer changes. No supporting-fact annotation is needed, which matters for RESQ, where none exists.
Demonstrations go through the same pruner. Each demonstration's story is pruned to its own chain before it is placed in the prompt. Separately, demonstrations are selected to match the pruned query's hop count.
Scope. We keep the study in text on purpose. What's "up" with vision-language models? and 3DSRBench show that visual spatial failures are tangled up with perception and viewpoint. Text lets us study composition and context competition without that confound. Extending the method to visual scene-graph contexts is left as follow-up work.
Baselines
Naive reference points. The do-nothing reference is the same reader on the full noisy story, using a standard fixed few-shot prompt with random demonstrations. A second reference is the reader on the full story with zero demonstrations. Every method runs with the same reader models, a mix of open-weight models at roughly 3B, 7B and 70B parameters. Every method uses the same demonstration pool and the same decoding settings.
Strongest learned model on StepGame. This is TP-MANN, which binds entities to spatial roles in a recurrent tensor-product memory. Its authors report 31.25% accuracy at $k=5$ hops on StepGame, against roughly 11–12% for relational networks. Because we evaluate on the corrected release, we will rerun TP-MANN there rather than quote the old figure. To count as beating it, the pruned language-model reader must exceed TP-MANN at $k \geq 5$ on the corrected data.
Strongest systems on SPARTQA and RESQ. These come from Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning. PISTAQ, a pipeline that extracts spatial triplets and passes them to a Prolog reasoner, reaches 96–97% on SPARTQA-Auto but degrades on real-world text. SREQA, a neural model with explicit entity-pair relation layers, reaches 69.5% on RESQ, against about 63% for BERT and about 60% for zero-shot GPT-3.5. The meaningful bar is SREQA's 69.5% on RESQ. On SPARTQA-Auto we treat PISTAQ as a ceiling that we do not expect to beat.
Symbolic baseline on StepGame. Map+ASP from the Advancing paper is rerun on the corrected data. The file reports no numeric accuracy for it, and we found no reported result on Lacuna for Map+ASP accuracy on corrected StepGame.
Iterative retrieval. IRCoT alternates between writing one reasoning sentence and using it as a query to fetch more text. We run it over the story's sentences, with the same reader and the same budget of reader calls. We found no reported result on Lacuna for IRCoT on StepGame.
Demonstration-side baselines. These are difficulty-based selection from PaRaDe, which picks the single demonstration the model finds least likely, and parallel grouping from Su et al., which runs subsets of demonstrations separately and combines their predictions with a Naive Bayes rule.
Ablations. The key ablation replaces the hop-conditioned score with EnComp-style independent sentence scores at the same kept-set size. This isolates the effect of conditioning on hops. Two further ablations prune only the story or only the demonstrations. All learned pruners get the same redaction-label budget and the same number of tuning runs.
Why not just…?
Why not just extract triplets and run a logic solver? This is the most serious threat. On synthetic StepGame and SPARTQA-Auto, PISTAQ and Map+ASP probably make the pruner unnecessary, and we will say so. The Disentangling paper shows that symbolic pipelines degrade on ambiguous real-world text. That is why RESQ and SPARTQA-Human are the headline evaluations. Our contribution stands only if pruning beats PISTAQ there and approaches SREQA's 69.5% on RESQ, or exceeds it. If the solver wins everywhere, we report that the reading-gap diagnosis holds but that a neural pruner adds nothing over symbolic extraction.
Why not just keep the sentences on the entity path between the two query entities? On synthetic StepGame, graph connectivity is almost an oracle, so this baseline is included as a near-oracle. It needs clean entity linking and coreference resolution, and real narratives in RESQ rarely provide either. The comparison between the path heuristic and the learned pruner on RESQ settles whether learning is needed.
Why not just use fewer or better demonstrations? PaRaDe shows that one well-chosen demonstration can beat four or eight random ones. If demonstration selection alone closes the gap, the story-distractor half of the claim fails. The two-way ablation of story pruning against demonstration pruning tests this directly. The interaction prediction, that $n^*$ rises only when the story is pruned, cannot be produced by demonstration selection alone.
Why not just run IRCoT? IRCoT keeps appending paragraphs, which reproduces the attention dilution we are trying to remove. Its intermediate queries can also drift after one bad retrieval. Under a matched budget of reader calls, it should trail the pruner on the irrelevant-branch noise type in particular.
Decisive Checks
Three results decide the proposal. First, oracle-chain accuracy must clearly exceed full-story accuracy at $k \geq 4$. If it does not, the reading-gap hypothesis is rejected. Second, pruning should shift $n^*$ rightward and restore the counterfactual attentiveness score. A flat accuracy-versus-demonstrations curve after pruning would also count as support, since it means extra demonstrations no longer hurt. Third, on RESQ, the hop-conditioned pruner must beat both the independent-score ablation and the entity-path heuristic. Accuracy is reported per hop count and per noise type. Bridge-sentence recall is also reported, against the oracle chain on StepGame and against spot-checked chains on RESQ. Pruner latency is reported as well.
References
- StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts
- Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark
- Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
- RiOT: Efficient Prompt Refinement with Residual Optimization Tree
- Efficient Encoder-Only Context Compression via Marginal Contribution Scoring
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
- Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- PaRaDe: Passage Ranking using Demonstrations with LLMs
- Naive Bayes-based Context Extension for Large Language Models