Intermediate Filters as Testable Hypotheses in Multi-Hop Retrieval

Systems that answer multi-step questions by retrieving documents work in rounds. A language model reads what it has found, writes down an intermediate conclusion ("the director of Inception is Christopher Nolan"), and uses that conclusion to search for the next piece of evidence. When the intermediate conclusion is wrong, every later search goes wrong too, and nothing in a free-text search tells the system that it has drifted. This proposal makes each intermediate step emit two things: a text query and a structured filter. The filter is a hard constraint on document metadata, such as an entity type, a time interval, or membership in a table row. The proposal then uses the filtered search as a cheap test of the intermediate conclusion. Suppose the best document that satisfies the filter is far worse than the best document overall, or almost nothing passes the filter. Then the hypothesis that produced the filter is probably wrong. The system can relax the filter or revise the step before the error propagates, rather than starving or hallucinating.

The opportunity sits between two mature communities that have not met. Database researchers have built fast indexes that answer "nearest neighbors subject to a predicate". These assume the predicate is handed over, fixed, with the query. Multi-hop retrieval methods refine their queries step by step, but only as free text. The obvious marriage, letting the model write a filter and handing it to a filtered index, mostly buys pruning. The claim here is narrower and, if true, more useful. The disagreement between filtered and unfiltered retrieval is an error signal that comes almost for free. It catches cascading mistakes that today require an extra language-model verifier call.

Core Claim

At each hop $t$, the reasoner proposes a sub-query embedding $\mathbf{q}_t$ and a predicate $P_t$ over document metadata. Retrieval returns two quantities. The first is the best filtered score $s_P = \max_{d \in \mathcal{D}, P_t(d)} \mathrm{sim}(\mathbf{q}_t, d)$. The second is the best unfiltered score $s_U = \max_{d \in \mathcal{D}} \mathrm{sim}(\mathbf{q}_t, d)$. Here $\mathcal{D}$ is the corpus and $\mathrm{sim}$ is the retriever's similarity. The system also counts $n_P$, the documents passing $P_t$ within a fixed similarity radius of $s_U$. We hypothesize that the consistency gap

$$\Delta_t = s_U - s_P$$

together with $n_P$ separates hops whose intermediate conclusion is correct from hops where it is wrong. A correct hypothesis about, say, the director's identity and the relevant date range should place the best-matching evidence inside the filter. A wrong one leaves the strongest evidence outside it. When the gap is large, the system steps down a relaxation ladder: it widens the time window, generalizes the entity type, and finally drops the predicate. It also feeds the out-of-filter top document back to the reasoner as a counter-hypothesis. What is new is the role of the predicate. It is a falsifiable claim about the next hop, checked by the index, rather than a user-supplied constraint or a pruning trick.

Grounding in Prior Work

The motivation comes from the multi-hop literature. Interleaving Retrieval with Chain-of-Thought Reasoning (IRCoT) uses each generated reasoning sentence as the next search query, gaining 11–21 points of retrieval recall. ITER-RETGEN re-retrieves with the whole previous draft. Both plateau after about two rounds. Beyond Static Retrieval (BDTR) shows that "bridge" documents get buried and that errors cascade. It answers with a second "slow thought" query aimed at bridge entities, plus a language-model verifier that reranks candidates. All three formulate hops only as text or graph walks. Our proposal keeps their loop and adds a structured, checkable side channel. Our cheap gap signal is a direct competitor to BDTR's verifier. MuSiQue constrains the design. Its 30-point F1 drop for single-hop models shows that loosening hop dependencies invites shortcuts, so relaxation must be triggered by evidence and logged, never applied by default. Layer-wise RAG avoids decoding by projecting middle-layer states into the retriever space. It is complementary and serves as our latency reference, at roughly 2.8 seconds per query versus more than 20 seconds for IterRetGen on 2WikiMultiHopQA.

The database side supplies the method. ACORN makes graph search tolerant of arbitrary predicates by routing through non-matching nodes. Compass coordinates a graph index with B+-trees using the "neighborhood passrate", the fraction of visited neighbors that satisfy the filter. Our $n_P$ statistic is a query-level relative of passrate, and Compass-style execution can produce $s_P$ and $s_U$ from one shared traversal. The filtered-ANNS survey warns that graph indexes fail when the query vector lies far from the filtered subset. Intermediate hops are exactly such queries, so we measure this rather than assume it away. Both systems were evaluated only with fixed, given predicates. Neither involves a reasoner that produces the predicate.

The closest attempts at the intersection are the benchmark papers. HybridQA shows that a staged table-then-text pipeline (HYBRIDER) reaches only 43.8% exact match, even though its modules are each about 88% accurate. Complex-TR poses temporal multi-hop questions whose constraints are literally scalar intervals, yet it retrieves by unfiltered dense ranking over contexts averaging 93,000 words. Neither generates predicates during reasoning, and neither uses predicate failure as feedback.

Research Program

We first attach metadata to existing corpora. Complex-TR comes with Wikidata time intervals and entity identities. For HybridQA, table rows and columns become attributes on linked passages, so "passages linked to rows where column X satisfies Y" is a predicate. For MuSiQue and 2WikiMultiHopQA, we attach Wikidata entity types and, where available, dates to Wikipedia paragraphs. This construction is our own and will be released. The reasoner is an instruction-tuned language model prompted, and in a second phase lightly fine-tuned, to emit at each hop a text query plus a predicate in a small typed grammar. The grammar covers equality on types, interval containment, and row membership. Fine-tuning uses predicates derived from gold decompositions.

The retrieval layer computes $s_U$, $s_P$, and $n_P$. For accuracy experiments it uses exact search: an exact pre-filtered scan for $s_P$ and exact unfiltered search for $s_U$. This keeps index approximation from confounding the reasoning result. A separate systems study replaces exact search with ACORN-1 and Compass. It measures whether intermediate-hop queries, which tend to sit far from the filtered region, lose filtered recall and thereby corrupt the gap signal. The controller compares $\Delta_t$ and $n_P$ against thresholds tuned on development data. It then accepts the hop, relaxes the filter by one step, or returns the out-of-filter top document to the reasoner with a prompt to reconsider. The number of hops is capped at three, following the plateau that ITER-RETGEN and BDTR report.

Baselines

The naive reference point is single-shot dense retrieval followed by reading. Its accuracy floor is shown by HybridQA's passage-only baseline at 19.1% exact match and HotpotQA's full-wiki baseline at 17.73 joint F1 (HotpotQA). We rerun it with our reader and retriever.

The strongest text-only competitor is IRCoT, since it is the most widely replicated interleaved method across MuSiQue, HotpotQA, and 2WikiMultiHopQA. The file reports its relative gains (up to 15 F1 points) but no absolute score. We reproduce it with the same reader model, retriever, corpus, hop cap, and prompt-tuning budget as our system. On MuSiQue, the absolute numbers in the file are L-RAG's 0.352 accuracy and BDTR's relative gain of 8.41% exact match and 6.73% F1. BDTR is the strongest error-correcting competitor, because its language-model verifier targets exactly the cascading failures our gap signal is meant to catch. We run it with the same reader and count its verifier calls in latency and token cost. On Complex-TR, the strongest reported numbers are 85.0% set accuracy for a PIT-tuned T5-large and 51.6% for GPT-4. On HybridQA, the strongest reported number is HYBRIDER at 43.8% exact match. Beating our own reproduction of IRCoT and BDTR at equal or lower latency is the main bar. Exceeding 43.8% on HybridQA and approaching 85.0% on Complex-TR without temporal pseudo-instruction tuning would count as beating the published systems. Matching BDTR's accuracy with no verifier calls would by itself be a positive result.

Three ablations isolate the key ingredient. The first, filter-as-text, writes the predicate into the query string with no hard filter. The second, hard-filter-with-fallback, applies the predicate and drops it only when nothing passes. The third, hard-filter-without-gap, uses the full relaxation ladder but triggers it only on emptiness, never on the gap. An oracle-predicate run, with gold predicates at every hop, bounds the achievable gain.

Why not just…?

Why not just write the constraint into the text query? Dense retrievers match surface semantics and are known to handle numbers and dates poorly. Complex-TR's reliance on unfiltered ranking over 93,000-word contexts, with GPT-4 at 51.6%, is consistent with this. The filter-as-text ablation measures the size of the effect directly. If it matches the hard filter, the predicate machinery is unnecessary and we will say so.

Why not just hard-filter and fall back to unfiltered search when the result is empty? This is the real threat to the proposal. Empty results are the most visible failure, and a fallback fixes them at zero cost. Our claim is that the damaging cases are not empty. A wrong hypothesis usually still admits some documents, just bad ones, so only the gap $\Delta_t$ reveals them. The comparison between hard-filter-with-fallback and the full system settles this. If the gap adds nothing beyond emptiness, the contribution shrinks to an engineering note.

Why not just retrieve a large pool, rerank, and let a long-context model sort it out? BDTR shows that bridge documents stay buried even when present in the pool. Long contexts also raise token cost. We include a top-100-plus-cross-encoder baseline at matched token budget to test this on our metadata-augmented corpora.

Why not just use an LLM verifier, as BDTR does? It probably works. The question is cost. Our gap signal comes from search statistics already computed during retrieval. We report accuracy against verifier calls and latency, and a hybrid that calls the verifier only on high-gap hops tests whether the two are complementary.

Decisive Checks

The first check comes before any end-to-end comparison and can kill the idea cheaply. MuSiQue's gold decompositions label each intermediate answer, so we can mark every hop as correct or wrong and compute the area under the ROC curve of $\Delta_t$ for detecting wrong hops. If the area is near 0.5 on MuSiQue and Complex-TR, the central claim is false.

Second, gains should concentrate where metadata is informative. We expect them on Complex-TR and HybridQA's passage-table-passage questions, smaller gains on MuSiQue, and none, or a controlled loss, on single-hop questions. The relaxation log should show that relaxation rarely enables shortcut answers. We check this on MuSiQue-Full's unanswerable contrast questions, where a system that loosens hop dependencies would wrongly produce answers.

Third, the systems study reports how much filtered recall ACORN and Compass lose on intermediate-hop queries relative to exact search, and whether that loss inflates false alarms from the gap signal. This directly tests the out-of-distribution concern from the filtered-ANNS survey in a setting it never examined.