Sparse "mixture-of-experts" models have many sub-networks, and a small router sends each token to only a few of them. This lets a model hold far more parameters than it uses per token. There is now good evidence that the extra parameters help with remembering facts but not with reasoning. Multi-step reasoning seems to need a minimum active width, meaning the size of the representation each token actually computes with, and adding experts does not raise that width. Multi-hop question answering needs both skills. It must recall facts ("Inception was directed by Christopher Nolan") and chain them together ("…and Nolan was born in London"). That raises a practical question nobody has answered: how should a builder with a fixed compute budget pay for the chaining step, given that sparse capacity is cheap?
We propose that an iterative retrieval loop can stand in for active width. A single forward pass that must connect $k$ facts needs width that grows with the connection problem. If each pass only has to resolve one hop, and the loop retrieves the next piece of evidence, then each pass is a lookup, and lookup is the regime where sparse models already scale well. We will measure the exchange rate between active width and the number of retrieval rounds. We will test whether it lets a narrow sparse model match a wide dense one on multi-hop question answering at equal total active compute, counted across all rounds. The result is a design rule for where to spend compute, and a direct test of the open question left by the lower-bound argument in the prior work.
Core Claim
Mixture of Parrots proves a communication-complexity lower bound for graph connectivity, which shows that shortest-path search requires a minimum hidden width. It also shows empirically that mixture-of-experts models match dense models of equal total size on TriviaQA and Natural Questions, yet flatline on shortest-path, MATH and GSM8k unless active width grows. That argument concerns a single forward pass over a problem held entirely in context.
Our claim is that the bound governs hops per pass, not hops per question. When an external loop breaks a $k$-hop question into $k$ single-hop passes, the width requirement drops to that of a lookup. Accuracy should then factor roughly as
$$ \mathrm{Acc}_k^{\text{iter}}(d, N) \approx \prod_{j=1}^{k} a_j(d, N), $$
where $d$ is active width, $N$ is total parameter count, and $a_j$ is the probability that pass $j$ both retrieves and correctly uses the evidence for hop $j$. The product form follows from assuming errors at different hops are independent, which is the compounding-error picture from the retrieval literature.
This formula makes predictions in both directions. In single-pass mode, accuracy should collapse once $d$ falls below a threshold that rises with $k$. In iterative mode, accuracy should track single-hop accuracy raised to the $k$-th power, and that single-hop accuracy improves with $N$ even at small $d$. If the prediction holds, the gap between sparse and dense models at matched active compute should close under iterative retrieval. If the gap persists, the bottleneck is not composition within a pass. It would instead be the synthesis of retrieved evidence, which would be a meaningful negative result.
Relation to Prior Work
Two lines of evidence motivate the proposal. Mixture of Parrots supplies the width bottleneck, and its authors explicitly leave open whether retrieval offsets it. Towards Principled Design of MoE under Memory and Inference Constraints fits a loss law in which adding experts under a fixed memory budget forces width and depth to shrink. That law is why "just add experts" cannot be the remedy. On the retrieval side, IRCoT alternates one reasoning sentence with one retrieval call. It shows a 3B model with this loop beating 175B GPT-3 with single-step retrieval, which is suggestive evidence that iteration substitutes for scale. However, it never controls width or sparsity.
The closest bridging work does not test this question. Open-RAG converts Llama-2-7B into a sparse model built from small fine-tuned adapter modules and reaches 63.3% exact match on HotpotQA. Its active width is essentially that of the dense base (7.0B of 7.8B parameters active), so it cannot tell us whether narrow sparse models are rescued. Chain-of-Experts adds sequential expert passes inside a layer and improves closed-book math loss from 1.20 to 1.12 at equal compute. We treat it as a competing way to buy serial computation, and in our setting that serial computation is internal rather than external. MergePRAG and L-RAG change how evidence enters a dense model: MergePRAG writes passages into weights through a hypernetwork, and L-RAG issues retrieval queries directly from middle-layer states. Neither varies active width, so both are orthogonal to our question. They serve as strong reference systems rather than as rivals to the claim.
Research Program
The core is a controlled grid that crosses three factors. The first is architecture: dense models at several widths, and sparse models at fixed active width with a growing number of experts, matched to the dense models on total parameters. The second is the number of hops $k$, from 1 to 6. The third is the inference mode, of which there are four:
- Closed-book single pass. Facts must come from the weights.
- Single pass with all evidence in context. This is the Parrots setting.
- Closed-book chain-of-thought. Serial computation happens without retrieval.
- Iterative retrieval. The loop follows the IRCoT style, with either an oracle retriever or a learned dense retriever.
Compute is always reported as active floating-point operations summed over every pass needed to answer.
The first stage is synthetic, so that width thresholds can be located exactly. We generate random knowledge graphs of entities and typed relations, and we pose $k$-hop composition queries over them. The facts either live in the training data, making them parametric, or sit in a retrievable store of one-fact passages. We also reuse the Erdős–Rényi shortest-path and phone-book tasks from Mixture of Parrots to anchor replication. The models are small enough to train from scratch at roughly 50M–400M active parameters, which keeps the full grid affordable.
The second stage uses natural data. We pretrain matched dense and sparse families at the largest affordable scale on a Wikipedia-heavy corpus. We then fine-tune each family identically for iterative retrieval, using supervision from decomposed questions. The models are evaluated on MuSiQue, whose composition-based construction yields gold hop counts of 2 to 4 and blocks shortcuts. We also evaluate on 2WikiMultihopQA and HotpotQA, and use PopQA/TriviaQA as single-hop controls. Stage two is where the factorization prediction meets real retrieval errors and distractors.
The main deliverable is the width-for-rounds exchange curve. At a fixed accuracy, it shows how much active width can be removed per added retrieval round, and whether total experts buy accuracy once iteration is available. A secondary deliverable is a small adaptive controller that picks the number of rounds per question from the model's own uncertainty, reusing DRAGIN-style signals. This tests whether the exchange rate can be exploited without knowing the hop count in advance.
Baselines
The naive reference point is standard single-step retrieval: one query from the question, top-$k$ passages concatenated, and one answer pass. It is run on every architecture in the grid with the same retriever and corpus. MergePRAG reports this baseline at F1 ≈ 40.51 on 2WikiMultihopQA with LLaMA3.1-8B, and Open-RAG reports base Llama2-7B at 5.9% exact match on HotpotQA. Any iterative arm must clear the single-step baseline of the same architecture, or the loop adds nothing.
The strongest competitor on the actual question is a wide dense model given the same iterative loop and the same total active compute. Concretely, it is the dense model whose per-pass cost times its number of rounds equals the sparse model's per-pass cost times its rounds. This is the baseline Mixture of Parrots implies should win on reasoning. Both are trained on identical data with identical token counts, fine-tuning recipes and hyperparameter search budgets, with the search budget counted in trials per model. The sparse model beats it if it reaches equal or higher MuSiQue answer F1 at equal summed active compute, with the gap outside seed variance over three seeds. We will also test at 3- and 4-hop questions specifically, where the width bound should bite hardest.
The strongest existing systems are MergePRAG, at F1 ≈ 79.34 on 2WikiMultihopQA (LLaMA3.1-8B), and L-RAG, at accuracy 0.352 on MuSiQue. DRAGIN reports F1 0.3931 on 2WikiMultihopQA with LLaMA2-13B, against 0.3076 for FLARE. These use far larger pretrained models than we can train, so we do not claim to beat their absolute numbers. Instead, we will run the retrieval strategies of IRCoT and DRAGIN on our own matched families. This separates "better loop" from "better architecture". We quote the published figures only to show where our absolute scale sits.
Two ablations isolate the key ingredient, which is the hop-per-pass structure. The first spends the FLOPs of $k$ rounds on a single pass instead, by giving the sparse model proportionally more active experts or depth. The second keeps $k$ rounds but hands each pass the full evidence set rather than the next hop's evidence. This keeps the extra compute but removes the decomposition. If either ablation matches the iterative arm, decomposition is not what matters.
Why not just…?
Why not just use a bigger dense model? At equal active compute, this is exactly our strongest baseline, so the experiment answers it directly rather than by argument. The existing evidence is only suggestive. IRCoT's 3B-beats-175B result and Open-RAG's 7B-beats-104B-Command-R+ result both show that the loop matters more than scale. Neither holds data or compute fixed. If wide dense models still win at matched summed compute, we will report that the width bound survives iteration.
Why not just use chain-of-thought without retrieval? This is a real threat. Writing intermediate steps also breaks a $k$-hop problem into single-hop passes, and in the closed-book case the facts could come from sparse parametric memory, which Mixture of Parrots shows scales with experts. We include closed-book chain-of-thought as a full arm of the grid. If it matches iterative retrieval on the synthetic graphs where every fact is parametric, then the operative ingredient is serial decomposition, not externality. The proposal's claim would narrow to "decomposition rescues sparse models, and retrieval is needed only for facts outside the weights." MuSiQue's long-tail entities would then show whether retrieval adds anything beyond that.
Why not just add more experts? Mixture of Parrots shows reasoning flatlining as experts grow at fixed active width. The constrained-design scaling law shows a 128-expert, top-8 configuration beating a 256-expert, top-16 configuration at equal sparsity. Our single-pass arms replicate this flatline, and the iterative arms test whether the same added experts become useful once each pass is a lookup.
Why not just use Chain-of-Experts internal iteration? It buys serial computation inside a layer and improved math loss at equal compute. However, it has never been tested on open-domain multi-hop questions, and its iteration count is fixed at training time rather than tied to the hop count. We train a Chain-of-Experts variant at matched active compute and run it both closed-book and inside the retrieval loop. This tells us whether internal relays and external rounds substitute for each other or stack.
Proposed Experiments and Decisive Checks
The first check is replication. In single-pass, in-context mode on shortest-path and $k$-hop graph queries, sparse models at fixed active width must flatline as experts increase while dense accuracy rises with width. Without that, the premise does not transfer to our setup.
The second check is the central prediction. With oracle iterative retrieval, the minimum active width needed for 90% accuracy should stop growing with $k$. $k$-hop accuracy should fall within a few points of single-hop accuracy raised to the $k$-th power, and single-hop accuracy should keep improving with total experts. We will plot measured against predicted accuracy per $k$.
The third check is the natural-data transfer. On MuSiQue, stratified by gold hop count, we will compare answer F1, supporting-passage recall and summed active FLOPs for matched sparse and dense families under the iterative loop. The central number is the sparse-minus-dense F1 gap at 4 hops, compared between single-pass and iterative modes. Our hypothesis predicts the gap shrinks sharply under iteration. A gap that stays constant would refute it.
The fourth check covers efficiency. We report wall-clock latency alongside FLOPs, because routing overhead and memory bandwidth can erase theoretical savings, as the Mixture of Attention Heads and Chain-of-Experts work notes. We also compare our cost against L-RAG's reported ~2.8 s versus >20 s per 2WikiMultihopQA query, as context for how expensive the extra rounds are in practice.
References
- Mixture of Parrots: Experts improve memorization more than reasoning
- Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Open-RAG: Enhanced Retrieval Augmented Reasoning with Open-Source Large Language Models
- Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
- MergePRAG: Orthogonal Merging of Passage-experts for Multi-hop Parametric RAG
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models