Multi-Hop Review Questions Whose Bridge Is a Group of Reviewers

Shoppers often ask questions that no single review answers and that no plain average of reviews answers either. Take "Does the battery hold up on multi-day hikes after the latest firmware update?" To answer it, a system must first find the reviewers who installed the update. Then, among only those people, it must find what they report about battery life on long trips. In standard multi-hop question answering on Wikipedia, the first step returns one bridging entity, such as a director's name, and that name keys the second search. In a review corpus, the first step returns a set of reviews. The correct second step is to search inside that set and count. Iterative retrieval systems do not do this. They match each new query against the whole corpus by text similarity. Their answer then blends the opinions of people who never met the condition with the opinions of people who did, and it usually drifts toward the overall consensus.

We propose cohort-conditioned retrieval. Each hop produces a verified group of reviews, which we call a cohort. The next hop is confined to that group. The final answer reports a conditional prevalence (for example, "9 of 14 reviewers who updated report the battery draining in under a day") with an uncertainty interval and the review IDs behind each count. We also build a benchmark of conditional and cross-product review questions. It applies the logic of MuSiQue's connectivity filtering to reviews: a question is kept only if ignoring the condition gives a measurably wrong answer.

Core Claim

The bridge object in multi-hop questions over reviews is a subpopulation, not an entity. Retrieval must therefore be scoped by membership, not by similarity. We predict three results. First, scoping the second hop to a verified cohort will cut the error of the estimated conditional proportion well below that of interleaved retrieval loops run on the same model. Second, it will reduce "target mismatch" on comparative questions, meaning the misattribution of one product's flaws to another. Third, an empty or tiny cohort gives the system an explicit, inspectable signal to back off or abstain. Forward-greedy loops lack such a signal.

Where This Sits

Two sources motivate the proposal. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT) alternates single reasoning sentences with retrieval. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (ITER-RETGEN) uses draft answers as the next query and plateaus after two to three rounds. Both show that iterative retrieval closes much of the gap one-shot retrieval leaves on HotpotQA, 2WikiMultihopQA and MuSiQue. Both also assume encyclopedic passages in which each hop has one true answer. Neither restricts later hops to the documents found earlier, and neither counts anything. Our method keeps their loop structure but changes what a hop returns and where the next hop searches.

The method draws on two further pieces of work. MuSiQue builds multi-hop questions by composing single-hop ones and filters out questions answerable by shortcuts. We reuse that recipe, but our shortcut is "ignore the condition and report the overall opinion." AmazonQA frames review question answering with one-shot BM25 retrieval and an answerability classifier. We keep answerability as a first-class output and derive it from cohort size instead of a separate classifier.

The review-synthesis side constrains the design. QQSUM is the closest prior work. It shows that standard retrieval-augmented generation (RAG, generating an answer from retrieved text) "collapses" onto one opinion. It fixes this with key-point retrieval, clustering, and quantified key points, and it releases the AMAZONKP benchmark. However, its retrieval is a single clustering step over one product's reviews, and its authors name target mismatch and multi-aspect sentences as unsolved. QQSUM quantifies how many people hold an opinion; we quantify how many people who satisfy a condition hold it, and we allow the condition to point into another product's reviews.

Other review systems aggregate statically within one product's review set. ASESUM extracts (aspect, sentiment, evidence) triples and scores support and contradiction. HERCULES and PAKPA count frequent opinions. Their reliance on frequency is exactly what washes out conditional, minority signals like the firmware example. We borrow PAKPA's sentence-level aspect parsing as an indexing step, but we apply it at query time inside a loop rather than offline.

Research Program

The system has four stages.

Decomposition. A language model rewrites the question into a chain of conditions $C_1,\dots,C_k$ and an outcome $O$. In the example, $C_1$ is "installed firmware update X", $C_2$ is "used on multi-day trips", and $O$ is "battery lasts". For cross-product questions, a condition may produce an entity instead. An example is "the replacement charger reviewers recommend", which redirects the next hop to that product's review pool. Every hop is tagged with a product ID so that evidence cannot cross between products.

Hop retrieval and verification. Reviews are indexed at clause level, so that "battery lasts all day, but the charging port broke" becomes two units sharing one review ID. This is the inexpensive answer to the multi-aspect problem QQSUM reports. For condition $C_j$, we retrieve candidates by combining BM25 and dense scores, restricted to reviews in the previous cohort $S_{j-1}$. A small verifier model, of the Qwen2.5-7B class used in ASESUM, then judges for each candidate whether the review asserts $C_j$. The verified review IDs form $S_j$. Because verification is per review and returns yes, no, or unclear, the cohort is an auditable set, not a soft context blob.

Back-off. If $|S_j|$ falls below a minimum size, the system relaxes the most specific condition, for instance from "update X" to "any recent update", and records that it did so. If even the relaxed cohort is too small, it abstains. This gives the loop a backtracking signal grounded in evidence, addressing the cascading-error problem of forward-greedy iterative RAG highlighted in Wang et al. (2025).

Counting and answering. Within the final cohort $S_k$, the verifier labels each review as supporting, contradicting, or silent on $O$. The reported quantity is

$$\hat p = \frac{n_{+}}{n_{+}+n_{-}},$$

where $n_{+}$ and $n_{-}$ are the numbers of reviews in $S_k$ that support and contradict $O$, respectively. We attach a Wilson score interval, because cohorts are often small and a normal approximation would give misleading intervals. The generator writes the answer from $\hat p$, the interval, the relaxation log, and two or three cited reviews per side.

The Benchmark

No benchmark in the file tests conditional or cross-product multi-hop questions over reviews. We will build one from review pools in AmazonQA and AMAZONKP, plus TripAdvisor hotels from SPACE. Hotels give natural conditions such as "travelling with children" or "stayed in winter". Products with 100 to several thousand reviews are sampled so that results can be plotted against pool size.

Construction proceeds in three steps. We mine frequent conditions and outcomes per category. We have every review labelled for each condition–outcome pair by an LLM, with humans checking a stratified sample of 1,500 labels to estimate label noise. We then compose 2-hop and 3-hop questions, including cross-product bridge questions.

The key filter follows MuSiQue. A question is kept only if the gold conditional proportion differs from the unconditional proportion $P(O)$ by at least 15 points and the gold cohort holds at least eight reviews. Otherwise a system that ignores the condition could score well by accident. The gold answer is the conditional proportion together with the supporting review IDs. A held-out set of questions will be written by people from real AmazonQA questions, to check that the synthetic composition does not create artifacts the method exploits.

Baselines

Naive reference. The do-nothing baseline is one-shot RAG as in AmazonQA: BM25 over the product's reviews with the raw question, the top ten snippets, and a language-model answer asked to estimate the proportion. A second naive reference is the unconditional proportion $P(O)$, computed from gold labels. Any method that does not clearly beat it is just reporting the overall opinion. This reference directly measures the shortcut the benchmark is built to punish.

Long-context reading. The strongest simple competitor is a long-context model given the entire review pool of the product or products. It is asked to identify qualifying reviewers and count, using the same generator model as our system. On small pools this may be very hard to beat. We therefore report it across pool-size bins and do not average it away.

Iterative RAG. The strongest existing multi-hop methods are IRCoT and ITER-RETGEN. We run both with the same generator, the same retriever and clause index, the same number of maximum rounds (three, matching ITER-RETGEN's plateau), and the same prompt-tuning budget on a shared development split. For reference, IRCoT reports retrieval recall gains of 11–21 points and QA F1 gains of up to 15 points over one-step retrieval on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC. ITER-RETGEN reports up to an 8.6% absolute gain over prior retrieval-augmented methods on four of six datasets. There is no reported result found on Lacuna for either system on review data or on any conditional-prevalence task.

Review-QA prior work. QQSUM-RAG is the strongest review question-answering system. On AMAZONKP its paper reports a 2.11x ROUGE-1 improvement over standard RAG and a 67.12% quantification improvement over PAKPA; the file gives only these relative figures, not absolute scores. We run QQSUM-RAG on our benchmark, using its released setup where possible.

Ablations. The central ablation removes cohort scoping: the full pipeline runs unchanged, except that each hop retrieves from the whole pool. This isolates the paper's single claim. Secondary ablations remove the per-review verifier (counting retrieved snippets directly), clause splitting, and back-off.

What counts as winning. Beating the baselines means a lower mean absolute error on $\hat p$ than both the unscoped ablation and the best of IRCoT and ITER-RETGEN, by a margin exceeding the label-noise estimate from the human audit, along with higher cohort F1 against gold review IDs. On AMAZONKP, which is mostly single-condition, the success criterion is not losing to QQSUM-RAG. We include that comparison to show the method does not trade away ordinary review question answering.

Why not just…?

Why not just put all the reviews in a long context window? For products with a few hundred short reviews, the whole pool fits, and this may work. We treat it as a real threat. The file offers indirect evidence that it degrades as pools grow: ASESUM reports that direct GPT-4o-mini summarization was its weakest system because of input-length saturation, and QQSUM documents opinion collapse in less structured pipelines. Neither result measures conditional counting, however. The pool-size curve settles the question. If long context matches our method below some pool size, we will report that crossover honestly and position cohort retrieval as the method for large pools and cross-product questions. Those are cases where the pools combined exceed what a model can read, and where per-review verification also provides attribution that a single long-context answer does not.

Why not just extract aspects offline into a table and run SQL? When the condition is already a column in the table, a join answers the question exactly. That case is in fact our method with the verifier run ahead of time. The weakness is coverage. The Wayfair pipeline in End-to-End Aspect-Guided Review Summarization at Scale drops aspects below frequency thresholds. RATION shows that aspect tags lose the rationales users need. Conditions such as "multi-day hikes" or "after update X" are unlikely to be in any fixed taxonomy. We will build this baseline with a strong LLM extractor and an oracle-generous taxonomy, then split benchmark questions by whether their conditions appear in it. We expect a tie on in-taxonomy questions and a win only on open-vocabulary ones, and we will report both halves.

Why not just run IRCoT or ITER-RETGEN on reviews? They are the right loop, and our method keeps their structure. Their known failure in this setting is architectural. They add retrieved text monotonically to one context and search the whole corpus at every hop, so reviewers who do not meet the condition re-enter at hop two. The unscoped ablation is exactly this scenario under matched settings, so if scoping does not matter, the ablation will show it directly.

Why not just prompt a ReAct-style agent? ITER-RETGEN reports that Self-Ask style agents use more calls and documents while aggregating evidence worse. More basically, an agent treats a tool's return value as a fact, whereas here the return value is a population that must be counted. We include a ReAct agent with access to the same scoped search tool. If it learns to scope and count on its own, that would show the value lies in the tool rather than the pipeline, which would still support the core claim.

Decisive Checks

Three results would falsify or confirm the idea. The first is the gap between the full system and the unscoped ablation on conditional-proportion error, reported separately for 2-hop and 3-hop questions. If it is within label noise, the core claim fails. The second is the target-mismatch rate on cross-product questions: the fraction of cited evidence whose review ID belongs to the wrong product, compared against QQSUM-RAG and IRCoT. The third is the calibration of abstention: on questions whose gold cohort is deliberately tiny, the system should abstain or widen its interval rather than report a confident number.

We will report cost in model calls per question for every system. Per-review verification is cheap but numerous, so it must be weighed against the 2–3 large-model rounds of the iterative baselines.