Learning Who Hears What: Curriculum-Trained Message Routing for Noisy Multi-Hop Question Answering

Questions like "In what city was the director of Inception born?" require multi-hop reasoning: the system must first find an intermediate "bridge" fact (the director is Christopher Nolan) before it can look up the final answer. Systems that do this well usually split the work across several language-model agents, one per sub-question or role, and wire those agents together in a fixed pattern decided in advance. Every agent then sees everything its predecessors retrieved, including the irrelevant passages. Each hop adds more distracting text, and language models are known to be thrown off by irrelevant context. Errors therefore compound with every hop. This proposal builds a system in which the agents learn, during training, which other agents' messages they should receive at each step. A curriculum that raises both the number of hops and the amount of distracting text teaches this routing.

The central bet is about timing and difficulty, not architecture alone. A router that decides who talks to whom can only learn something useful when its choices change the outcome. On clean, easy questions, extra messages do no harm, so the router has no reason to cut any connections. On very noisy, deep questions, an untrained system almost never succeeds, so the reward carries almost no signal about which connections mattered. The router can only learn in a middle band of difficulty, and a curriculum that tracks learning progress should find and stay in that band. If the bet is right, learned sparse routing should beat both fixed multi-agent pipelines and single noise-robust readers. The margin should grow with hop count and distractor density and vanish when there are no distractors.

Core Claim

Compounding distraction in multi-hop retrieval can be framed as a message-routing problem: what goes wrong is which earlier evidence reaches which later reasoning step. Routing can be learned with reinforcement learning, but only under a two-axis curriculum over hop depth and distractor density, with the schedule chosen by learning progress. The system is trained once on MuSiQue. It should then reduce the share of irrelevant text in each agent's context and improve answer and supporting-fact accuracy on held-out 3- and 4-hop questions under injected noise. A sparse topology hand-matched to each question's decomposition should not achieve this.

Several papers motivate the claim. Improving Multi-Agent Debate with Sparse Communication Topology shows that in math debate, a fixed sparse ring of agents outperforms all-to-all communication. Dense communication spreads noise and pushes agents toward agreeing too early, so who sees what is a lever on noise. DyTopo rebuilds a sparse communication graph every round. Each agent writes a short "what I need" and "what I offer" description, and edges form where the two match. It reports gains on code and math, but its matching threshold is fixed, nothing is trained, and it was never tested on retrieval over a document corpus. The multi-hop agent systems ReAgent and BELLE take a different approach. ReAgent adds backtracking over hard-coded roles, and BELLE debates which reasoning operators to use within a fixed debate structure. Both send evidence along wiring that never changes. On the curriculum side, EVO-RAG moves a single query-rewriting agent from an exploration phase to a precision phase, but there is only one agent and nothing to route. Learning Progress Driven Multi-Agent Curriculum supplies the scheduling method we reuse. It found that a seemingly harder setting (more agents) is not reliably harder to learn from, and it schedules by measured learning progress instead of raw return. We expect the same non-monotonicity along our noise axis. The proposal therefore differs from DyTopo by training the routing, from ReAgent and BELLE by making the wiring change per step, and from EVO-RAG by applying the curriculum to the communication structure rather than to a single policy.

Research Program

The system has a planner, a pool of hop workers, and an assembler, all built on one open 8B instruction model with separate low-rank adapters. Each worker handles one sub-question, retrieves passages, and writes a short evidence message. Before a worker reads anything from its peers, it writes a "need" descriptor and every other worker writes an "offer" descriptor, as in DyTopo. Worker $i$ receives worker $j$'s message only if

$$s(q_i, k_j) = \cos(W q_i, W k_j) > \tau,$$

where $q_i$ and $k_j$ are frozen sentence embeddings of the need and offer descriptors, $W$ is a small learned projection, and $\tau$ is a threshold. Raw retrieved passages never flow between workers. Only the evidence messages that pass routing do, which caps how much distractor text any worker can inherit.

Training uses Group Relative Policy Optimization (GRPO), the reinforcement learning method AgentPO uses (AgentPO). GRPO compares groups of sampled rollouts for the same question. The parameters trained are the projection $W$ and the descriptor-writing adapter. The reward is answer F1 plus supporting-fact F1, minus a small penalty per routed token so the router has a reason to cut connections. Worker reasoning adapters stay frozen in the main condition. This isolates routing as the thing being learned.

The curriculum is a grid of cells indexed by hop count $h \in \{2,3,4\}$, taken from MuSiQue's own composition annotations, and injected distractor count $d \in \{0,4,8,16\}$. Distractors are extra non-supporting paragraphs mixed into each worker's retrieval results. Cells are sampled in proportion to recent learning progress, defined as

$$\mathrm{LP}(h,d) = \left| \bar{R}_t(h,d) - \bar{R}_{t-\Delta}(h,d) \right|,$$

the absolute change in mean reward $\bar{R}$ on that cell over the last $\Delta$ updates. We use the change in reward rather than a critic's error because GRPO has no critic. Self-Paced Learning for Latent Variable Models and the finite-sample analysis in A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula motivate why easy-before-hard ordering helps optimization. Our specific prediction is sharper and falsifiable. The learned schedule should spend most of its time in middle-noise cells ($d=4$ or $8$), not at the easy corner. Routing gradients should be weakest at $d=0$ even though accuracy is highest there.

We will also log a mechanistic quantity for every worker at every step: its noise exposure, the fraction of tokens in its context that come from non-supporting paragraphs, measured against MuSiQue and HotpotQA gold supporting facts. The paper's causal story requires that learned routing lowers exposure and that lower exposure predicts fewer bridge-entity errors.

Baselines

The naive reference point is IRCoT (Interleaving Retrieval with Chain-of-Thought Reasoning), which alternates reasoning sentences with retrieval and appends every retrieved document to one growing context. It runs on the same 8B model, retriever, and corpus. It reports up to 15 F1 points over one-shot retrieval across its datasets, but the file gives no absolute MuSiQue number. Beating it means a higher answer F1 on MuSiQue-Ans that holds at every distractor level.

The strongest existing multi-agent competitor is ReAgent. With GPT-4o it reports an F1 of 0.795 on HotpotQA, against 0.661 for O1 and 0.696 for O3. We will re-run ReAgent's roles on our 8B model with the same retriever, capped at the same mean number of model calls as our system. ReAgent averages 29 calls per question, so we also report it uncapped. Beating it means higher F1 at equal calls, and at least parity uncapped. BELLE reports a 7.6% improvement on 4-hop MuSiQue, but the file gives no absolute value. We will re-run BELLE under the same budget. No reported result was found on Lacuna for any absolute MuSiQue F1 of these systems. Human performance on MuSiQue-Ans, 78.0 answer F1 and 93.9 support F1, serves as the ceiling (MuSiQue).

The strongest single-agent noise-robust competitor is Ext2Gen (Aligning Extraction and Generation for Robust RAG). It trains one reader to extract relevant sentences and answer jointly, and reports 0.557 versus 0.473 for the default Llama3.1-8B on its noisy suite. We will retrain it on the same MuSiQue training split with the same injected distractors and match its GPU-hours to ours. It will then be wrapped in IRCoT-style iteration so that it can handle multiple hops.

Untrained DyTopo, transplanted to retrieval with its original fixed threshold, isolates the value of training the router. No reported result was found on Lacuna for DyTopo on multi-hop QA.

Three ablations isolate the proposal's ingredients. The first trains the same router on a uniform mix of all cells, which isolates the curriculum. The second uses a fixed linear easy-to-hard schedule, which isolates learning-progress scheduling from ordinary ordering. The third, the most important, is an oracle fixed topology: workers are wired exactly along each question's gold decomposition graph from MuSiQue's annotations, so no learning is needed. If the oracle wiring matches learned routing, the contribution reduces to "follow the decomposition."

Why not just…?

Why not just train a single reader to ignore noise, as Ext2Gen does, and iterate it? This is the most serious threat. Ext2Gen already shows that training on noisy retrieval helps a single model. Our argument is that in a single accumulating context, every hop's distractors stay visible to every later hop, and soft attention cannot fully exclude them. Routing removes them from the context entirely. The experiments settle this directly. If iterated Ext2Gen matches our system at $d=16$ on 4-hop questions under equal compute, the routing claim fails and we will report it that way.

Why not just use a fixed sparse topology? The sparse-debate paper shows that a static ring already beats dense communication. The oracle-decomposition baseline is a stronger version of this alternative, since it is sparse and correct by construction. Learned routing can only win if useful connections depend on what was actually retrieved, for example when a worker's retrieval fails and a sibling's evidence turns out to hold the bridge entity. We will break results down by MuSiQue's six graph shapes and by whether the first-hop retrieval succeeded, so it is visible where any advantage comes from.

Why not just pick a topology per query, as AMAS does, or train a collaborator inside a fixed pair, as AgentPO does? AMAS chooses one of four precomputed graphs before execution begins. On HotpotQA and MuSiQue it cannot react to a bad retrieval mid-chain, but this is our hypothesis, not evidence. We will therefore include an AMAS-style selector over four candidate graphs, one of which is the oracle chain. AgentPO argues that topology search is not worth its combinatorial cost. Our search space is a single learned similarity per pair of workers, not a search over graphs. We will report the router's training cost against AgentPO's 500-sample regime so readers can judge the trade-off.

Proposed Experiments

Training uses the MuSiQue-Ans training split. Evaluation covers MuSiQue-Ans and MuSiQue-Full (the latter includes unanswerable questions, so refusals count), the HotpotQA distractor setting, 2WikiMultiHopQA as a transfer target, and MultiHop-RAG with its Inference and Null query types. For reference on MultiHop-RAG, GPT-4 reaches 0.56 accuracy with retrieved chunks and 0.89 with ground-truth evidence (MultiHop-RAG), so that gap measures the cost of noise. Metrics are answer EM and F1, supporting-fact F1, MuSiQue-Full answerability F1, routed tokens and model calls per question, and the per-worker noise-exposure measure.

Four checks are decisive. The first is the interaction test. The gain over iterated Ext2Gen should be near zero at $d=0$ and grow with both $h$ and $d$. A flat gain would mean routing is not what is helping. The second is the mechanism test. Learned routing must lower noise exposure relative to IRCoT and the fixed topologies, and within each system, exposure at hop $k$ should predict bridge-entity errors at hop $k+1$. The third is the curriculum test. The learning-progress schedule must beat both uniform mixing and the linear schedule on held-out 4-hop, $d=16$ questions, and its visit counts should concentrate in middle-noise cells as predicted. If it lingers at the easy corner instead, our account of why curriculum helps routing is wrong even if accuracy improves. The fourth is the oracle test. Learned routing must at least match gold-decomposition wiring on clean questions and exceed it on questions whose first-hop retrieval failed. Otherwise the honest conclusion is that multi-hop systems need correct decomposition, not learned communication.