Syntax-Guided Query Formation for Dynamic Multi-Hop Retrieval

Retrieval-augmented language models answer questions by pulling passages from a corpus and reading them. They struggle with multi-hop questions, where the fact needed for the second step ("where was Christopher Nolan born?") only becomes visible after the first step is resolved ("who directed Inception?"). The most economical current systems do not write a new search query at each step. They watch the model as it generates and, when it looks uncertain, build a query out of the context words it is attending to most. We argue that this "pick the words the model attends to" step fails in a predictable, linguistically specific way. In multi-hop questions, the word the uncertain token actually depends on is often grammatically linked but far away in the text. Relative clauses, appositives, and nested possessives push it out of reach ("the birthplace of the director, whose early work was shot on film in London, of the 2010 heist movie"). Language modeling has shown for years that such syntactically distant positions are systematically hard for sequence models. We propose to build retrieval queries from the dependency tree, meaning the grammatical graph linking each word to the word it modifies, instead of from raw attention.

In two sentences: when a dynamic retrieval system decides to search mid-generation, it should choose query words by grammatical closeness to the uncertain token, not by attention weight alone, because attention leaks to nearby distractors exactly when the relevant word is syntactically distant. We test this by sorting existing multi-hop benchmarks by how far the "bridge" relation stretches in the dependency tree, and by building controlled paraphrases that lengthen that distance while keeping the meaning fixed.

Core Claim

The multi-hop reasoning gap is partly a long-distance dependency problem, and it shows up at a specific place: query formation. DRAGIN decides when to retrieve from next-token entropy and attention influence. It decides what to retrieve by taking the top-$n$ context tokens by attention from the triggering token. Revisiting the Case for Explicit Syntactic Information in Language Models found that about a quarter of word positions have their grammatical head outside a local window. At those positions, 4-gram perplexity was about 60% higher, and ten times more data did not close the gap. Bisazza's keynote reports the same weakness for recurrent models on nested, non-local structure. Our claim has three parts. First, the accuracy of attention-based dynamic retrieval falls as the tree distance between the triggering token and the bridge entity grows. Second, reweighting query-token selection by dependency-tree distance recovers much of that loss at almost no extra cost. Third, the benefit comes from syntax specifically, not from general locality or from extra computation.

The mechanism is one scoring rule. Let $t$ be the token at which retrieval is triggered and $a_{t,i}$ the attention weight from $t$ to earlier token $i$, as used by DRAGIN. Let $d(t,i)$ be the number of edges between $t$ and $i$ in the dependency tree over the question plus the generated prefix. Query tokens are the top-$n$ tokens under

$$s_i = a_{t,i}\, \exp(-\lambda\, d(t,i)),$$

where $\lambda \ge 0$ is tuned on a development split. Setting $\lambda = 0$ recovers DRAGIN exactly. Replacing $d(t,i)$ with the linear distance $|t-i|$ gives a locality-only control. The tree is built over the question and all generated text, so a word inside a relative clause and the noun that clause modifies count as neighbors, even when many words separate them.

Relation to Prior Work

The work that uses structure for multi-hop questions does so offline, before any retrieval happens. Question Decomposition with Dependency Graphs parses questions into graphs of logical operations on the BREAK benchmark. It is evaluated only on decomposition exact match, never with a retriever, and its graphs follow the BREAK schema rather than grammatical syntax. Interpretable AMR-Based Question Decomposition segments a semantic graph into sub-questions and reaches 81.17 F1 on the HotpotQA distractor setting. It relies on hand rules for 2-hop shapes, works over ten pre-selected paragraphs, and inherits errors from its semantic parser. We do something different from both. We do not decompose the question up front. We intervene inside the retrieval loop at each triggered search, and we use surface grammatical trees over generated text as well as the question. These papers motivate the idea that question structure matters. The parser-error cascade reported in the AMR paper constrains our design: it is why we test a parser-free variant.

The dynamic retrieval line supplies the method we modify. IRCoT uses each chain-of-thought sentence as the next query and gains 11–21 points of retrieval recall. ITER-RETGEN alternates retrieval and generation in a similar way. CoRAG trains a model to write explicit sub-queries and raises MuSiQue recall from 29.0% to 47.1%. DRAGIN is the closest system and the one we change. We keep its trigger and replace only its query-token selection. For the structure itself we draw on the syntax literature. Dependency paths already serve as filters in relation extraction (Zhang et al.) and event detection (Nguyen et al.), where the path between two entities discards intervening clauses. We apply the same intuition to choosing search terms. DEPPROBE shows that a probe of about 127k parameters can read labeled trees from a frozen encoder's middle layers, with a mean labeled attachment score of 60. That result motivates our parser-free variant. The file gives no evidence that it transfers to decoder-only models, so we treat that as an assumption to test.

Research Program

The first component is a diagnostic. For every question in 2WikiMultihopQA, HotpotQA, and MuSiQue-Ans, we parse the question with a Universal Dependencies parser. Universal Dependencies provides the annotation scheme and treebanks. We then compute bridge distance: the tree distance and the linear distance between the phrase naming the unresolved relation and the phrase naming its anchor entity. We identify these phrases using the gold supporting facts and, for MuSiQue, the gold single-hop compositions. We then run single-round RAG, FLARE, DRAGIN, and IRCoT, and report accuracy and retrieval recall within bins of bridge distance. This tells us whether any method's failures concentrate at syntactically distant bridges, echoing the local-versus-distant split in the language modeling study. It also tells us whether methods that write their own queries (IRCoT) avoid the drop that attention-selected queries (DRAGIN) suffer.

The second component is a controlled stress test. MuSiQue questions are built from single-hop pieces, so we can write meaning-preserving paraphrases that insert relative clauses, appositives, or parentheticals between the bridge relation and its anchor. We generate these with a language model and check them by hand on a sample for fluency and semantic equivalence. The result is a set of question families that hold content fixed and vary only syntactic distance. This is the direct test of the claim, because it removes confounds such as distant-bridge questions simply being rarer or harder entities.

The third component is the intervention: the scoring rule above, in two versions. In the external-parser version, a standard UD parser is run incrementally over the question and generated prefix. In the probe version, a DEPPROBE-style probe is trained on English UD over one middle layer of the generating model, chosen by development-set probe accuracy. The probe version needs no separate parser, and its tree comes from the model's own representations. That matters for conversational or poorly formed generated text, where external parsers are brittle.

Baselines

The naive reference points are no retrieval and single-round retrieval. In single-round retrieval, the question is embedded once, the top-$k$ passages are fetched, and the model answers. It shows how much any iterative method adds. MultiHop-RAG shows its ceiling: even with reranking, Hits@10 reaches only 0.7467. We run it with the same retriever and the same $k$ as every other system.

FLARE and DRAGIN are the dynamic retrieval baselines. DRAGIN is the strongest direct competitor because our method changes exactly one of its components. For LLaMA2-13B on 2WikiMultihopQA, DRAGIN reports an F1 of 0.3931, against 0.3076 for FLARE (DRAGIN). We rerun DRAGIN from its released settings with the same model, retriever, corpus, and triggering thresholds. We give it the same development-set budget we give our method, used to tune $n$ and its thresholds. Beating it means a statistically significant F1 gain across seeds and bootstrap resamples on the full test sets, with no more retrieval calls. It also means a larger gain in the top bridge-distance bins and on the stress-test paraphrases.

IRCoT is the strongest method that writes its own queries. On our question it serves as a ceiling-style comparison, not a direct rival, because it pays for extra generation at every step. IRCoT reports up to 15 F1 points over one-step retrieval across its datasets. We run it with the same model and retriever and report F1 together with tokens generated and wall-clock latency. CoRAG is a trained reference point. It reports MuSiQue recall of 47.1% and over 10 exact-match points above IterDRAG and Search-o1-32B (CoRAG). Our method is training-free, so we cite CoRAG for context and do not claim to beat it.

Three ablations isolate the key ingredient. The linear-distance kernel tests whether the gain comes from syntax or from simply favoring nearby words. A random tree with the same number of edges tests whether any sparse structure helps. Parser-versus-probe trees test sensitivity to parser errors. The claim survives only if the real tree beats both the linear and the random controls.

Why not just…?

Why not just let the model write the query, as IRCoT and CoRAG do? This is the most serious alternative. A fluent model that writes "Where was Christopher Nolan born?" has already resolved the long-distance link in its own generation. If the diagnostic shows IRCoT is flat across bridge distance while DRAGIN drops, then written queries solve the problem. Our contribution then narrows to recovering that robustness at the token budget of dynamic retrieval, and we will report it that way. If IRCoT also drops on the stress-test paraphrases, the syntactic account is broader than query formation. Either outcome is informative, and the accuracy-per-generated-token comparison decides which case holds.

Why not just use a larger model? IRCoT shows that a 3B model with iterative retrieval beats a 175B model with one-step retrieval. Larger models also still struggle: Llama2-70B scores 0.32 on MultiHop-RAG even with gold evidence. Scale is nonetheless the natural way attention could learn to reach distant heads. We therefore run the diagnostic on two model sizes and check whether the distance-bin slope flattens with scale. If it does, the method matters mainly for small models.

Why not just decompose the question up front with an AMR or logical-form parser? QDAMR and the dependency-graph decomposer show that structure helps decomposition. However, both are limited to fixed schemas or 2-hop rules, and neither interacts with retrieved text. In our setting, the distant dependency often sits in the generated reasoning, not only in the question, and offline decomposition cannot see it. We will add a decompose-then-retrieve baseline that runs few-shot sub-questions through the same retriever. This tests whether in-loop selection beats up-front splitting on 3- and 4-hop MuSiQue questions.

Why not just prefer nearby tokens? Recency weighting is cheaper than parsing. The linear-distance ablation is designed to rule it out. On the stress-test paraphrases, linear locality should actively hurt, because the inserted clause is nearer to the trigger than the true anchor.

Proposed Experiments

We evaluate on 2WikiMultihopQA, HotpotQA, MuSiQue-Ans, IIRC, and StrategyQA, matching DRAGIN's suite, using LLaMA2-13B for comparability and Llama-3.1-8B-Instruct as a newer model. Metrics are exact match and F1, retrieval recall of gold supporting passages, retrieval calls per question, generated tokens, and latency. The file reports no per-dataset DRAGIN numbers on HotpotQA, IIRC, or StrategyQA, so we will reproduce those rather than quote them.

Two checks decide the project. The first is the slope of accuracy against bridge distance on the controlled paraphrases. DRAGIN should decline, and the syntax-weighted version should decline much less. If DRAGIN is flat, the premise is wrong and we stop. The second is the ablation ordering: real tree above linear kernel and random tree, with the probe-derived tree close to the external parser. If only the external parser helps, the method is limited by parser availability. If the probe matches it, structure read from the model's own hidden states is enough to fix what raw attention gets wrong.