Train the Planner in a Fictional World, Read in the Real One

Language models trained on synthetic, rule-generated puzzles about fictional people and places can get better at real multi-hop questions, such as "Who directed the film that won the award the year this author was born?" Recent work shows that the skill of chaining facts transfers even when no fact overlaps. It also shows two persistent costs. Supervised training on synthetic reasoning traces makes the model copy the generator's sentence templates. And reinforcement learning on synthetic data still falls short of training on real data. We propose to find out which part of the reasoning carries that remaining gap.

The proposal is to split question answering into two jobs. The first is planning a decomposition tree: breaking a question into sub-questions, nested so that independent branches can be answered separately and then combined. The second is answering the leaf sub-questions against real text. We train the planning part only in synthetic worlds, where the generator's logic program gives a free, checkable answer at every node of the tree. We then measure, on real benchmarks that come with human-verified decompositions, how much of the remaining synthetic-to-real gap belongs to planning and how much to leaf reading. Our bet is that planning transfers almost fully and the gap sits mostly in leaf reading. If so, cheap real single-hop data should close most of what remains.

Core Claim

This follows a lesson from robotics sim-to-real transfer. Decomposition helps transfer when the parts trained in simulation are the parts the simulator models faithfully. The robotics evidence comes in two forms:

- Pre- and post-contact policy decomposition splits manipulation into an approach phase and a force-regulation phase and changes the control interface. With this split, zero-shot real-world success rises from 0–13% to 87%.

  • ODYSSEY keeps simulation-trained whole-body control at the bottom of an LLM-planned hierarchy.

Neither work decomposes language queries. In text, a fictional world models the structure of composition exactly but models human phrasing and real documents poorly. We therefore claim the following. A small model trained with reinforcement learning to emit and execute an explicit decomposition tree, rewarded at every node by synthetic ground truth, will match or beat outcome-only training on real multi-hop QA. The advantage should concentrate on questions whose structure branches, such as comparisons, intersections and counts. We also claim the residual gap will be attributable mostly to leaf answering rather than to tree construction.

Prior Work and Its Role Here

The closest work is Learning from Synthetic Data Improves Multi-hop Reasoning, which we call Kabra et al. throughout. It is both our motivation and our main competitor.

  • What it does: it trains Qwen3 models with outcome-only reinforcement learning (GRPO) on PhantomWiki and related fictional generators. The model reasons in a free linear chain of thought.
  • What it reports: relative F1 gains of 56% to 131% on real benchmarks for a 0.6B model, and generalization to deeper hop counts.
  • What it finds about supervised training: fine-tuning on synthetic traces overfits to templates.
  • What it leaves open: it never makes the plan explicit and never uses the per-node ground truth that its own generators contain. It therefore cannot say where its remaining gap to real-data training lives.

The method we reuse comes from Reasoning over Hierarchical Question Decomposition Tree (RoHT).

  • What it does: it builds a tree with the composite question at the root and atomic sub-questions at the leaves. A scheduler routes each node, and an aggregator combines child answers.
  • What it reports: 54.4 EM on MuSiQue and 46.45 EM on modified KQA Pro. Its ablation shows a drop when the tree is flattened into a chain.
  • What it does not do: it is trained and run on real data and never asks whether the planning ability could come from synthetic data.

Two further papers constrain the design.

  • Is a Question Decomposition Unit All We Need? shows that human-written decompositions give large gains, for example GPT-3 on DROP rising from 52.97 to 78.16 F1. It also reports that automatic decomposers fall short of human ones on completeness and dependency structure. Our planner has to clear that bar, so we measure decomposition quality directly rather than only end accuracy.
  • MuSiQue builds questions bottom-up from single-hop questions and filters out shortcut-solvable compositions. This gives us two tools. First, gold real-world decompositions for attributing the gap. Second, a contrast setting with unanswerable questions (MuSiQue-Full) that tests whether a model actually executes its hops.

The perceptual sim-to-real papers do not enter the experiments. Sim VQA and CoC-VLA answer either in one shot or along a single linear perception-to-planning chain. We deliberately stay in text, where the gap can be measured cleanly, before making any multimodal claim.

Research Program

The output format. One policy model writes a decomposition tree. Each node holds a sub-question and a typed operation: bridge, compare, intersect or count. The model answers leaves from the provided context, fills child answers into parent nodes, and produces the root answer. Leaves are answered before parents, but sibling branches never see each other's answers. That separation is the error-isolation property RoHT argues for.

The reward. PhantomWiki's logic programs define the true value of every intermediate relation. That lets us give a verifiable reward at each node. A node is credited if its sub-answer matches the true value of some sub-goal on a valid path to the root. Matching by value, not by wording, keeps the template-copying problem Kabra et al. found with supervised training from re-entering through the back door.

The training objective adds the outcome reward to the mean node reward:

$$R = r_{\text{root}} + \lambda \cdot \frac{1}{|T|}\sum_{v \in T} r_v$$

  • $r_{\text{root}}$ is the outcome reward.
  • $T$ is the set of nodes the model wrote.
  • $r_v \in \{0,1\}$ is the node-level reward.
  • $\lambda$ is a weight tuned on a synthetic validation split only.

Gap attribution. On MuSiQue, which comes with gold single-hop decompositions, we run a two-by-two swap:

  • {synthetic-trained, real-trained} planner × {synthetic-trained, real-trained} leaf answerer;
  • separately, each answerer given gold sub-questions;
  • separately, each planner scored for tree agreement with the gold decomposition.

This tells us how much of the end-to-end shortfall against real-data training comes from the plan and how much from reading. If reading dominates, a final stage adds a small amount of real single-hop supervision, drawn from MuSiQue's own single-hop component questions, to the leaf answerer only. The test is whether this cheap fix recovers most of the real-data ceiling.

Baselines

Do nothing. The naive reference is the same base models, Qwen3-0.6B and a roughly 7B model as in Kabra et al., prompted for direct answers and separately for linear chain of thought. They see the same retrieved or distractor paragraphs and receive no training. Every gain is reported relative to this.

Strongest synthetic-only method. This is Kabra et al.'s outcome-only GRPO with linear chain of thought on PhantomWiki. It answers our exact question: can synthetic data alone teach real multi-hop reasoning? We re-run it ourselves to ensure:

  • the same base checkpoints, synthetic generator configuration and number of training questions;
  • the same number of policy-gradient steps and rollouts per question;
  • a matched hyperparameter search budget, with selection on a synthetic validation split for every method so that no method sees real data during tuning.

The file reports only relative improvements of 56% to 131% F1 for this method; no absolute per-benchmark result is found on Lacuna for it. Our re-run is therefore the reference. Beating it means a statistically reliable F1 gain on MuSiQue-Ans across three seeds, together with no loss on HotpotQA and 2WikiMultihopQA. We would treat a gain of a few F1 points concentrated on branching question types as meaningful. A gain only on chain-shaped questions would suggest a format effect rather than structure.

Real-data reference. This is a model trained the same way on MuSiQue's training split, which defines the ceiling. RoHT's reported 54.4 EM on MuSiQue serves as an external anchor for what a real-data tree method achieves, though with a different model.

Ablations isolating our ingredients.

  • Tree format with outcome-only reward (λ = 0) isolates whether the dense synthetic node reward matters.
  • Node reward with the tree flattened into a chain isolates the tree structure.
  • Supervised fine-tuning on gold synthetic trees tests whether our gains survive the template problem or are caused by avoiding it.

Why not just…?

Why not just use Kabra et al.'s outcome-only reinforcement learning, which already transfers? It is the real threat. It may already learn implicit decomposition, and the explicit tree may add nothing. The λ = 0 ablation and the head-to-head comparison settle this. If our method only ties, the gap-attribution experiment still yields a finding the prior work cannot provide: where its residual gap lives. We would report the null result plainly.

Why not just prompt a strong model to decompose zero-shot, with no synthetic training? For large models this may suffice. Is a Question Decomposition Unit All We Need? reports, however, that automatic decomposers fall short of human ones on completeness and dependencies, and its largest gains come from human decompositions. We include prompted decomposition at both model sizes. We expect it to fall short mainly at 0.6B. If it matches our method at 7B, we will scope the claim to small models.

Why not just train on a little real data and skip the synthetic world? For HotpotQA-style data this is often the practical answer, and the real-data reference measures it. Our claim concerns settings where gold decompositions or intermediate answers are expensive. Our final stage tests the cheaper middle path: a synthetic planner plus a small amount of real single-hop leaf data. That arm wins only if it approaches the full real-data ceiling with far less real annotation.

Why not just fine-tune on gold synthetic trees, since they are free? Kabra et al. found that supervised fine-tuning on synthetic traces overfits generator templates. Our supervised-tree ablation checks whether that holds for trees as well. If it does not, supervised training is the simpler method and we would recommend it.

Proposed Experiments and Decisive Checks

Training and evaluation data. All methods train on PhantomWiki. Evaluation covers:

  • HotpotQA in the distractor setting;
  • 2WikiMultihopQA;
  • MuSiQue-Ans and MuSiQue-Full;
  • CofCA, a counterfactual set that removes factual shortcuts;
  • deeper-hop synthetic splits up to nine hops, for depth generalization.

Metrics. We report:

  • answer EM and F1;
  • for MuSiQue-Full, the paired answerable/unanswerable score, as a test that the model actually executes its hops;
  • tree agreement with MuSiQue gold decompositions;
  • leaf accuracy given gold sub-questions.

All results are broken down by question type: bridge, comparison and compositional.

Decisive check 1: structure. Gains over Kabra et al.'s method should be larger on comparison and multi-branch questions than on pure bridge chains. Uniform gains would mean the tree is acting as a formatting aid rather than a structural one.

Decisive check 2: attribution. In the swap experiment, a synthetic-trained planner paired with a real-trained answerer should come close to the all-real system. The reverse pairing should not. If planning carries most of the gap instead, the core hypothesis is wrong, and synthetic worlds would need richer linguistic variety in how sub-questions are phrased.

Decisive check 3: shortcuts. On MuSiQue-Full, the tree model should lose less than the chain-of-thought baseline when moving from answerable to contrast questions. That would show its decompositions are executed rather than bypassed.