How Much of the Multi-Hop Gap Is Grammar?

Language models that answer questions by looking things up do much worse when the answer needs several chained facts. "In what city was the director of Inception born?" requires finding the director first and then the birthplace. The standard explanation is that retrieval or reasoning breaks: the system cannot fetch the second fact before it knows the first, or it cannot chain facts at all. There is a second, untested explanation. Multi-hop questions are almost always written as nested grammar: a relative clause inside a possessive inside a prepositional phrase. Separate work shows that models fail on grammatically tangled questions even when the logic is within their reach. We propose to measure how much of the multi-hop gap comes from this grammatical load rather than from missing evidence or failed chaining.

The method is to build minimal pairs. These are versions of the same multi-hop question that keep the facts, entities, hop structure, and length fixed, and vary only how hard the sentence is to parse. We then run the pairs through readers given the correct paragraphs, through one-shot retrieval, and through iterative retrieval systems. If grammatical load explains a meaningful share of failures, and if that share persists under iterative retrieval, then a cheap rewrite that only reorders the grammar becomes a credible intervention. The rewrite needs nothing but text access to the model, so it also works with commercial APIs. If the effect vanishes once retrieval is iterative, that is equally useful: it would tell the field that existing iterative methods already absorb the parsing problem.

Core Claim

Syntactic complexity can be measured with Dependency Locality Theory (DLT). DLT scores a sentence by how far apart grammatically linked words sit and how many unresolved grammatical expectations a reader must hold open at once. Our claim is that this measure predicts multi-hop failure beyond what hop count, question length, and retrieval recall predict. We further claim the effect has two separable parts. One is a reading cost, where the model mis-parses the chain even with the right paragraphs in hand. The other is a retrieval cost, where the nested question is a worse search query for the first hop.

The motivating evidence comes from two sides. Syntactic Blind Spots found that high DLT scores predict LLM failures on math word problems. Rewriting failed problems into simpler dependency structures, with semantics manually confirmed to be preserved, raised Granite-7B accuracy on MultiArith by 15.5 points. That work is closed-book and single-question, and it never touches retrieval or multi-hop chains. Higher-order Comparisons of Sentence Encoder Representations found that disagreement between middle layers of BERT tracks a parse-depth score called Yngve depth (correlation 0.66). This suggests nested syntax places a measurable strain on transformer representations.

On the multi-hop side, MuSiQue reports roughly a 30-point gap between the best model and humans once shortcuts are removed, and attributes it to connected reasoning. Our design tests that attribution directly instead of assuming it.

Why MuSiQue makes the controlled study possible

The main confound is that grammatical nesting and hop count rise together: each added hop usually adds an embedded clause. MuSiQue avoids this by construction, because it composes each question bottom-up from single-hop questions joined by bridge entities. Having those components lets us recompose the same chain in several grammatical forms while the underlying chain stays fixed. The forms are:

  • the original nested form;
  • a center-embedded form that raises DLT storage cost;
  • a right-branching form that keeps each head next to its dependents;
  • a two-sentence form that introduces the bridge with a pronoun ("Inception was directed by someone. In what city was that person born?").

Every form contains the same content words. Function words are padded or trimmed so that token counts match within a small band. This matters because That Looks Hard and Is this Sentence Difficult? both show that raw length dominates complexity measures unless it is binned out.

2WikiMultiHopQA provides a second, template-generated source with explicit evidence triples, which permits the same recomposition.

One point needs care. The two-sentence form must not hand the model a solution plan that the nested form hides. To keep this honest, we include a fifth condition: an explicit sub-question decomposition with placeholders. Comparing the two-sentence form against this fifth condition separates "easier grammar" from "the plan was given away."

Research Program

We first build the minimal-pair set, MuSiQue-Syn, by template recomposition plus light LLM smoothing. Each item is scored with length-normalized DLT integration cost, DLT storage cost, parse depth, and maximum dependency distance, parsed with a Universal Dependencies parser. Two annotators verify a random sample for semantic equivalence. Parsing errors are a known weakness on coordination and parataxis (Exploring diachronic syntactic shifts), and the templated questions are short. Even so, we report how sensitive the conclusions are to the choice between two parsers.

We then evaluate every variant under three conditions:

  • Reading only. The gold supporting paragraphs plus MuSiQue's distractors are given to the model. This isolates reading cost.
  • One-shot retrieval. The question alone is the search query. This adds first-hop retrieval cost.
  • Iterative retrieval. The system is IRCoT (Interleaving Retrieval with Chain-of-Thought Reasoning), which uses each generated reasoning sentence as the next query, or DRAGIN (DRAGIN), which decides when to retrieve from token uncertainty and attention.

Readers span open models at roughly 7B, 13B, and 70B parameters, so we can see whether the effect shrinks with scale.

The analysis is a mixed-effects logistic regression of answer correctness on syntactic cost, with each underlying chain as a random effect:

$$\text{logit}\,P(\text{correct}_{iv}) = \alpha_i + \beta_h h_i + \beta_\ell \ell_{iv} + \beta_s s_{iv}.$$

Here $i$ indexes the underlying chain and $v$ its grammatical variant. $\alpha_i$ is a per-chain intercept that absorbs everything about the facts themselves. $h_i$ is hop count, $\ell_{iv}$ is token length, and $s_{iv}$ is the normalized DLT cost. Because $\alpha_i$ is fixed within a chain, $\beta_s$ is identified only from variation across grammatical variants of the same facts. That is exactly the quantity we want. We report $\beta_s$ separately under each condition and the fraction of the human–machine gap closed by moving from the hardest to the easiest variant.

Finally, we test an intervention: DLT-minimizing rewriting at inference time. Unlike the recomposition, which uses gold components, the rewriter sees only the raw question. It parses the question, produces candidate reorderings that move each dependent next to its head, keeps the candidate with the lowest DLT cost, and checks entity preservation. The rewritten question then feeds an unchanged IRCoT pipeline. This deliberately departs from the template-matching rewrite of Syntactic Blind Spots, which requires a pool of already-solved problems to copy structure from. Our rewriter targets the DLT score directly.

Relation to existing decomposition work

Three lines of work decompose multi-hop questions, and all treat the question's grammar as something to discard rather than measure:

  • QDAMR splits a semantic graph, specifically Abstract Meaning Representation (AMR), into fluent sub-questions and reaches 81.17 F1 on HotpotQA's distractor setting. Its rules are specific to two-hop questions, and it ignores dependency length.
  • Question Decomposition with Dependency Graphs parses decompositions as token-level graphs of logical operations, not grammatical ones. It never connects them to retrieval.
  • Query Decomposition for RAG treats sub-queries as arms of a bandit, a model that balances trying new options against exploiting known good ones, chosen under a context budget.

Our proposal does not decompose at all in its main condition. It asks whether keeping the question whole but grammatically easy recovers part of what decomposition recovers. That in turn tells us how much of decomposition's known benefit is really a parsing benefit.

Baselines

The naive reference point is the original MuSiQue question fed to the same reader and retriever with no rewriting. Every comparison is within-item against this. The strongest published reading-side number on Lacuna is the Select+Answer model's 47.3 F1 on MuSiQue-Ans, against 78.0 for humans, both reported in MuSiQue. We use this only to calibrate the gap, since our readers differ.

The strongest iterative competitor is IRCoT. The file reports only relative gains for it, 11–21 points of retrieval recall and up to 15 F1 (Interleaving Retrieval with Chain-of-Thought Reasoning). No absolute MuSiQue F1 for IRCoT was found on Lacuna, so we rerun it ourselves with the same retriever, corpus, reader, and step limit. For dynamic retrieval, DRAGIN reports 0.3931 F1 on 2WikiMultiHopQA with LLaMA2-13B, against 0.3076 for FLARE (DRAGIN). We reproduce that setting first and require our rerun to land near 0.39 before adding the rewriter.

The decomposition competitor is LLM-prompted sub-question decomposition with placeholders, given the same model and prompt-tuning budget as the rewriter. QDAMR is included on two-hop HotpotQA items, where its rules apply.

The ablation that isolates the key ingredient is a length-matched LLM paraphrase that is not told to minimize DLT. It is generated with the same model and filtered for entity preservation. Beating it means the DLT-minimizing rewrite improves F1 over generic paraphrase by a margin whose bootstrap confidence interval excludes zero. Beating IRCoT means rewrite-plus-IRCoT exceeds plain IRCoT under identical retrieval budgets.

Why not just…?

Why not just prompt the model to decompose the question? This is the main threat. If explicit decomposition recovers everything the grammatical rewrite does, and more, then syntax is merely a side door to planning. The fifth condition and the decomposition baseline settle it. If the two-sentence form closes most of the gap between the nested form and full decomposition, then the gain is grammatical. If it closes little, we will report that decomposition's benefit is mostly about planning. Syntactic Blind Spots suggests unguided generation does not reliably fix structural misalignment, but that evidence comes from math problems and may not transfer.

Why not just use length or a readability score? Length is the strongest single predictor of complexity in That Looks Hard. That is exactly why our variants are length-matched and the regression includes $\ell_{iv}$. If $\beta_s$ is indistinguishable from zero once length is in the model, the syntactic hypothesis fails. We also fit a Flesch-Kincaid predictor as a direct competitor to DLT.

Why not just scale the model or rely on iterative retrieval? Both may absorb the effect. Iterative retrieval in particular replaces the nested question with generated sentences after the first step. That is why we estimate $\beta_s$ separately at each model size and under IRCoT. A slope that shrinks toward zero under IRCoT and at 70B would be a clean negative result, showing the problem is confined to one-shot, small-model pipelines.

Decisive Checks

The study hinges on four outcomes:

  • Reading condition. Under gold paragraphs, is $\beta_s$ negative and significant with chain identity, hop count, and length held fixed? If not, the reading-cost half of the claim is false.
  • Retrieval condition. Does first-hop recall@k of the one-shot retriever differ across grammatical variants of the same chain? This tests the retrieval-cost half.
  • Iterative condition. Does the slope survive under IRCoT? If it does not, any rewriter is unnecessary there.
  • Intervention. Does the DLT-minimizing rewriter beat length-matched paraphrase on raw MuSiQue questions, where no gold components are available?

A positive answer to the first and last, with a nonzero slope under IRCoT, would justify grammar-aware rewriting as a black-box addition to multi-hop pipelines. Any other pattern still yields a quantified attribution of the multi-hop gap, which no paper in this literature provides.