Presence-Only Distillation of Multi-Hop Search Behavior

Multi-hop questions ("In what city was the director of Inception born?") can only be answered by a system that searches, reads, and then searches again for an entity it did not know at the start. The best recipe for giving small models this skill is to have a large teacher model produce search-and-reason trajectories, keep the ones that reached the right answer, and fine-tune the small student on them. This recipe quietly treats every trajectory the teacher did not produce, or produced and got marked wrong, as a negative example. Multi-hop data is actually presence-only: a successful path shows that one route works, while a failed or missing path shows very little. Ecologists face the same problem when mapping where a species can live from sightings alone. Their standard answer is maximum entropy modeling, which finds the least-committal distribution that still matches what the observations force you to believe.

We propose to distill multi-hop search behavior the way species distribution models are fit. The target is the teacher's own distribution over search trajectories, reweighted by a small exponential tilt. The tilt is chosen so that the expected values of a handful of interpretable path features match their averages over successful paths, and the distribution stays as close as possible to the teacher everywhere else. The student is then trained on this tilted distribution rather than on success-filtered samples. The claim is that this transfers what makes trajectories succeed, not the particular successful strings. It should therefore generalize better to harder, shortcut-resistant multi-hop benchmarks. It also extracts training signal from questions where the teacher never succeeded, which success filtering throws away.

Core Claim

Success-filtered distillation is one extreme point in a family of targets. In our notation, $\tau$ is a full search trajectory: the sequence of queries, retrieved passages, and intermediate statements. $p_T(\tau \mid x)$ is the teacher's sampling distribution for question $x$, and $f(\tau)\in\mathbb{R}^d$ is a low-dimensional vector of path features. We fit the distribution $q$ that is closest to the teacher in relative entropy, which is the "maximum entropy relative to a background" formulation used in ecology, subject to matching feature expectations on successful paths:

$$q_\lambda(\tau \mid x) \;\propto\; p_T(\tau \mid x)\,\exp\!\big(\lambda^\top f(\tau)\big), \qquad \mathbb{E}_{q_\lambda}[f] \;\approx\; \hat{\mathbb{E}}_{\text{success}}[f] \pm \beta .$$

Here $\lambda\in\mathbb{R}^d$ holds the learned feature weights and $\beta$ is the regularization tolerance. This exponential form is the standard dual solution of a moment-constrained entropy problem. Success-filtered fine-tuning corresponds to replacing $\exp(\lambda^\top f)$ with the indicator of answer correctness. In ecological terms, that is a model which puts all probability on the observed sighting locations, the Dirac-spike overfitting failure listed among MaxEnt's known limitations.

The features are meant to capture transferable search behavior. They include whether each new query contains an entity introduced by the previous hop's evidence, lexical overlap between the query and the original question (a shortcut indicator), novelty of retrieved passages relative to earlier hops, hop count, answer-type agreement, and the Wikipedia popularity of the queried entity (a teacher-bias indicator). Because $d$ is small, $\lambda$ is a readable table of which behaviors predict success, much as MaxEnt habitat studies report variable contributions. For example, Jiang et al. fit MaxEnt to 732 presence points and attribute 61.82% of predictive power to climate variables. Ren et al. isolate precipitation of the coldest quarter as a 32.2% contributor.

The ecological idea we borrow most directly is target-group background. In species distribution modeling, spatial sampling bias toward roads and cities is corrected by drawing background points from the same survey effort that produced the sightings. The analogue here is to use the teacher's full set of samples, successes and failures together, as the background. The tilt then learns what separates successful paths from paths the teacher typically produces, not what separates them from arbitrary text. The teacher's habits, such as favoring popular entities or copying question words into queries, cancel out rather than being inherited.

Relation to Prior Work

The work that motivates this proposal is the distillation-based multi-hop line. Auto-RAG synthesizes 10,000 retrieval-decision trajectories with Qwen1.5-32B and fine-tunes Llama-3-8B on them. It reaches 48.9 F1 on 2WikiMultihopQA, against 21.0 for standard retrieval and 21.6 for Iter-RetGen. This is exactly the success-filtered, sequence-level recipe we generalize. We keep its teacher, student, and data pipeline and change only the training target. MuSiQue supplies the evidence that this matters. It shows that HotpotQA permits single-hop shortcuts, and that a single-hop model loses 30 F1 points on MuSiQue-Ans relative to HotpotQA. A student that imitates teacher strings trained on HotpotQA-style data may therefore imitate shortcuts, and a feature-level target can penalize those shortcuts explicitly.

Two further lines use entropy in multi-hop retrieval, and both play a different role from ours. DRAGIN reads the next-token entropy and attention weights of a frozen model to decide when to retrieve and what to query. BEE-RAG adds a balancing term to the attention softmax so that attention entropy stays constant as retrieved context grows. In both, entropy is a diagnostic or a normalizer inside one model at inference time. Neither distills anything, and neither treats unobserved reasoning paths as presence-only evidence. Making Slow Thinking Faster is the closest distillation-plus-entropy work. It replaces low-entropy reasoning steps with a skip token before fine-tuning and reports 35–57% token savings on math benchmarks. It uses entropy as a pruning filter on closed-book traces, whereas we use entropy maximization as the training objective over retrieval trajectories. It is a compatible, orthogonal compression step.

IRCoT and AutoRefine supply the method we build on and the strongest non-distillation competitor. IRCoT shows that intermediate reasoning sentences make good next-hop queries, and that a 3B Flan-T5-XL with interleaved retrieval beats 175B GPT-3 with single-step retrieval. This is why a small student is a reasonable target. AutoRefine trains the policy directly with reinforcement learning on a correctness-plus-retrieval reward. Its reward is an indicator of success, so it inherits the same presence-only problem through a different optimizer.

We also acknowledge that the tilted form above is mathematically the same family as maximum-entropy inverse reinforcement learning and reward-weighted regression. We do not claim the equation is new. What is new is how the problem is posed and what is borrowed from ecology: successes treated as presence-only data, the teacher's own samples used as a target-group background, a regularization tolerance tuned against over- and under-constraint, and the use of the tilted distribution as a distillation target for a small retrieval agent.

Research Program

Sampling. The teacher (Qwen1.5-32B, matching Auto-RAG) samples $K=8$ trajectories per training question from HotpotQA and 2WikiMultihopQA. It uses the Auto-RAG retrieval-planning format over a dense index of the December 2018 Wikipedia dump used by AutoRefine. Every trajectory is kept, with its answer-correctness label, and each is scored with the feature vector $f$.

Fitting the tilt. We fit $\lambda$ by the convex dual with an L1 penalty, as ecological MaxEnt does, sweeping $\beta$. This step is cheap: it is a $d$-dimensional convex problem over a few hundred thousand stored trajectories.

Building the target. Every stored trajectory receives the normalized weight $\exp(\lambda^\top f(\tau))$ within its question. This includes the trajectories from questions where all $K$ samples failed; such paths still get high weight if they show the behaviors that predict success elsewhere. We resample by these weights to build the student's fine-tuning set, at the same number of training tokens as the success-filtered baseline.

Students. We train Llama-3-8B-Instruct, and Flan-T5-XL as the capacity-stressed student.

Analysis. Beyond accuracy, we report the fitted $\lambda$ table, retrieval recall of gold supporting passages per hop, and the entropy of the student's first-hop query distribution. The last lets us check the intended mechanism: the student should be more diverse on features the constraints leave free and sharp on features they pin down.

Baselines

The naive reference points are single-pass retrieval with the same student and retriever, and the same student prompted with IRCoT-style interleaving without fine-tuning. Auto-RAG reports 21.0 F1 on 2WikiMultihopQA for standard retrieval. We expect our reproduction to land near that value. If it does not, this signals a pipeline difference we must resolve before comparing anything else.

The strongest existing distillation method is Auto-RAG's success-filtered fine-tuning. Its reported result is 48.9 F1on 2WikiMultihopQA for a Llama-3-8B student, and 44.3 averaged over six benchmarks against 35.5 for Iter-RetGen. We rerun it with our retriever, the same $K$ teacher samples per question, the same number of student training tokens, and the same hyperparameter search budget. Our own reproduced number, not the paper's, is the bar. For MuSiQue, we found no reported result on Lacuna for an Auto-RAG-style distilled student. The comparison there rests entirely on our reruns, with the MuSiQue human score of 78.0 F1 as a ceiling.

The strongest non-distillation competitor is AutoRefine-style reinforcement learning applied directly to the student. AutoRefine reports gains of 0.083 on 2WikiMultihopQA and 0.045 on MuSiQue over its baselines. The file gives no absolute scores. We run it on the same questions with its answer-plus-retrieval reward, capped at the same teacher-plus-student compute as our pipeline. DRAGIN's 0.3931 F1 on 2WikiMultihopQA with LLaMA2-13B is cited for context only, because it addresses retrieval timing with white-box access rather than training a student.

Three ablations isolate the key ingredients. The first uses a uniform background, meaning the tilt is fit against generic trajectories rather than the teacher's own samples, and tests whether target-group bias correction matters. The second drops the shortcut and popularity features. The third sweeps $\beta$ from near zero, which approaches success filtering, to large, which approaches the untilted teacher.

We count the proposal as beating the success-filtered baseline if it improves MuSiQue-Ans answer F1 by at least 2 points, averaged over three seeds with non-overlapping bootstrap intervals, while not losing more than 1 point on the in-distribution 2WikiMultihopQA development set.

Why not just…?

Why not just sample more teacher trajectories and fine-tune on all the successful ones? This is the most serious threat. Keeping several distinct successes per question already adds diversity, and it may capture most of the benefit. We handle it by making multi-success filtering, with all correct samples out of $K$, the main baseline rather than single-success filtering. We also sweep $K$ from 4 to 32 for both methods. If the gap closes as $K$ grows, the MaxEnt target is only a sample-efficiency trick, and we will report it that way. The one advantage filtering cannot match by construction is signal from questions where every teacher sample failed. Measuring student accuracy on held-out questions of that kind is a direct test.

Why not just add an entropy bonus or label smoothing to ordinary fine-tuning? A uniform entropy bonus widens the student's distribution on every feature, including the ones that decide success, such as carrying the bridge entity forward. The constrained objective widens it only where the success data are silent. We test this by tuning the entropy-bonus weight until the student's first-hop query entropy matches ours, then comparing accuracy and per-hop gold-passage recall at equal entropy. If the entropy bonus matches us there, the feature constraints are not doing work.

Why not just train the student with reinforcement learning on answer correctness, as AutoRefine does? Reinforcement learning optimizes the same success indicator, so it faces the same presence-only problem. It also typically sharpens the policy rather than preserving alternatives. It is still a strong method, and it may win in-distribution. We therefore compare at matched compute. We also test the combination, using a MaxEnt-distilled initialization followed by reinforcement learning, because the two approaches may be complementary rather than rivals.

Why not just use a bigger model? IRCoT's result that a 3B model with interleaved retrieval outperforms 175B GPT-3 with single-step retrieval shows that search behavior, not scale, is the bottleneck in this regime. Deployment cost is the reason to distill at all. We include the teacher itself as an upper reference and report what fraction of the teacher-to-student gap each method closes.

Proposed Experiments

The decisive experiment trains on HotpotQA and 2WikiMultihopQA and evaluates out of distribution on MuSiQue-Ans and Bamboogle. MuSiQue's single-hop composition was built to defeat shortcuts, so it is where a feature-level target should pay off if it pays off anywhere. We report answer F1, supporting-passage recall per hop, and number of retrieval calls. On MuSiQue-Full we additionally measure how often the student abstains correctly on the unanswerable contrast questions. A student that learned shortcuts should answer those confidently.

A second check tests whether the $\beta$ sweep traces the U-shape seen in species distribution modeling. Under-constraint should drift toward teacher imitation, and over-constraint should collapse toward success filtering, with the best out-of-distribution score in between. The absence of such a curve would suggest the tilt is not acting as a regularizer at all. A third check inspects the fitted $\lambda$. We expect a negative weight on question-query lexical overlap and a positive weight on bridge-entity carryover. If instead the weights reward popularity or overlap, the teacher's successes are themselves shortcut-driven, and that is a finding about the benchmarks.

The proposal is falsified if multi-success filtering at $K=32$ matches the MaxEnt target within seed noise on MuSiQue and Bamboogle, and the zero-success-question subset shows no gain. In that case, treating multi-hop supervision as presence-only data does not change what a distilled student learns.