Recommending Which Peer Should Hint: Collaborative Filtering for Reasoning Rescue

When a small language model fails a specialist problem, such as a physics question that hinges on a theorem it never absorbed, the natural fix is to show it part of a stronger model's solution. It mostly does not work. Off-Trajectory Reasoning gave failing models partial reasoning traces from stronger models on math problems, and solve rates stayed under 9.2% unless the hint gave away the answer. This proposal treats that failure as a matching problem that recommender systems already know how to solve. A recommender is a system that guesses which item a person will like from the choices of people with similar tastes. Here the question is which guide's hint will rescue which receiver model on which problem. We would record, across many receiver models, guide models and problems, whether a hint actually rescued the receiver. We would then predict rescue for new problems from the outcomes of similar receivers on similar problems, where "similar problems" means problems that need the same domain theorem.

The core bet is that rescue depends on fit between guide and receiver, not on guide strength. Collaborative filtering has repeatedly found that the most useful advisor is a matched peer rather than the global best. Crowd proxies picked by agreement on a few calibration items predicted a user's taste far better than random workers (A Crowd of Your Own). A scholar's unfamiliarity with jargon was best predicted by peers with similar publication backgrounds (Personalized Jargon Identification). If the same holds for reasoning hints, the guidability ceiling partly reflects choosing guides badly, not a limit of the receiver, and a cheap selection rule that needs no training could lift it.

Core Claim

Given a fixed one-hint budget, choosing the guide by collaborative prediction of rescue will produce a clearly higher rescue rate than handing over the strongest model's trace. It will also cause less harm when the hint is wrong. The collaborative prediction combines rescue outcomes from receivers that fail on the same calibration problems with outcomes on problems that share a required theorem. The new element is the object being recommended: a partial reasoning trace from a particular guide, chosen for a particular receiver on a particular problem. The supervision is an observed rescue outcome. Neither consumer clicks nor textual relevance plays that role.

Relation to Prior Work

The closest bridge between the two directions is RecLM. It conditions a language model on the histories of neighboring users to write user profiles for news and movie recommendation. Its neighbors serve to predict consumption. We use neighbors to predict whether a hint repairs a reasoning failure, and we need no teacher model to write profiles. Personalized Education with Ranking Alignment Recommendation aligns question sequences across students with similar learning goals, but only inside simulated tutoring environments with discrete question IDs. Our "students" are real models and our "items" are natural-language traces.

Two papers motivate the gap. Off-Trajectory Reasoning measured the guidability ceiling, but always took the guide as given. It also found that strong models are easily derailed by injected text: AM-Thinking-32B recovered from distraction only 33.4% of the time. TheoremQA found that open models score below 15%, and that supplying theorem definitions "did not provide a significant boost". The missing ingredient is how to apply the theorem, which a peer's worked trace carries and a dictionary definition does not.

Two papers constrain the design. Thinking in a Crowd shows that reasoning models rationalize misleading context: Qwen3-8B falls from 96.22% to 77.03% on SciAux with two misleading snippets. A badly chosen hint is therefore not neutral, and we must measure harm as well as gain. n-Musketeers found that pooling expert signals uniformly hurts on easy tasks: Reasoning Gym Logic drops from 96.88% to 82.81%. That argues for per-query selection over always-on mixing.

Two lines of work supply the method. From An automatic weighting scheme for collaborative filtering we reuse two ideas. The first is leave-one-out learning of item weights, motivated by its finding that only about 2% of items carry most of the signal. The second is its asymmetric notion of similarity. From taste-matching we reuse calibration-set agreement as the definition of receiver similarity. CoRe trains models to rescue one another, but only within a closed pool and only when some member succeeds first. Our method needs no training and can draw from any trace library, including traces of look-alike problems when no guide solved the target.

Research Program

The first step is building a rescue log, an interaction table of three-way outcomes. We take about eight open receiver models from 1.5B to 8B parameters and about eight guides, from the same families up to 32B, with both reasoning-tuned and plain chat models. Problems come from TheoremQA, which labels each of its 800 questions with one of over 350 theorems and spans math, physics, EE&CS and finance. MATH-500 is added for comparability with Li and Goyal, and GPQA and MedMCQA for domains where knowledge rather than calculation is the bottleneck.

For every failure by a receiver $r$ on a problem $p$, and every guide $g$ that solved $p$, we inject a prefix of $g$'s trace with the answer stripped. The prefix is a fixed fraction of the trace, following the Off-Trajectory protocol. We record whether $r$ then succeeds, using several samples per triple. We also inject hints on problems the receiver already solves, to log harm.

The second step is the predictor. Receiver similarity $\text{sim}(r,r')$ is weighted agreement of success and failure patterns on a held-out calibration set. The weights on calibration problems are learned by the leave-one-out conditional exponential objective of Jin et al., so that uninformative problems that everyone solves or everyone fails drop out. Problem similarity $\text{sim}(p,p')$ is one when two problems share a TheoremQA theorem label, and embedding cosine otherwise. The predicted rescue probability is a neighborhood average:

$$\hat{s}(r,g,p)=\frac{\sum_{(r',p')\in N(r,p)} \text{sim}(r,r')\,\text{sim}(p,p')\,s(r',g,p')}{\sum_{(r',p')\in N(r,p)} \text{sim}(r,r')\,\text{sim}(p,p')}.$$

Here $s(r',g,p')\in\{0,1\}$ is an observed rescue outcome, and $N(r,p)$ contains the $k$ most similar receiver–problem pairs with logged outcomes for guide $g$. The system hands over the hint from the guide with the highest $\hat{s}$, or no hint if the top score falls below a threshold tuned on validation. This abstention option targets the rationalization harm.

The third step addresses the case where every guide failed. CoRe offers nothing there. We instead recommend a solved trace of a look-alike problem $p'$ that uses the same theorem, scored by the same predictor with $p'$ standing in for $p$.

All evaluation is on held-out problems. Calibration problems are disjoint from test problems, and theorem labels on test problems are used only through the similarity term. A variant that predicts the theorem with a classifier removes even that dependence.

Baselines

The naive reference point is no hint at all. On failure-conditioned triples this gives a rescue rate of zero by construction, apart from resampling luck. We measure that luck by resampling the receiver with the same extra token budget, so that gains are not just retries.

The strongest existing reference is the protocol of Off-Trajectory Reasoning: hand over the trace of the strongest guide that solved the problem. On math it reports guidability solve rates below 9.2%. We rerun it on our exact problems, receivers and prefix length, so that numbers are compared like for like.

A second strong competitor is a learned rescue classifier with no neighborhood structure. It is gradient-boosted trees over features of receiver, guide and problem, including model size, family, problem embedding and guide trace length, trained on the same rescue log with the same tuning budget. This is the ablation that isolates the collaborative ingredient. If it matches our predictor, the gain comes from logging rescues, not from collaborative filtering.

Further ablations remove the learned calibration weights (uniform weights), remove theorem-based problem similarity (embedding only), and remove receiver similarity (problem neighbors only). An oracle that picks the best guide per triple in hindsight gives the ceiling, and a hint containing the final answer gives a leakage reference.

All methods choose from the same candidate traces under a budget of one hint. Beating the strongest-guide baseline means a statistically significant gain in rescue rate on held-out problems in at least TheoremQA and one non-math domain, with harm on already-solved problems no higher than that baseline's.

Why not just…?

Why not just always use the strongest model as the guide? This is exactly what Li and Goyal tested, and solve rates stayed under 9.2%. Their finding that stronger models are more fragile hints that strength and compatibility come apart. Still, this is the main threat. If the hindsight oracle's rescue rate is barely above the strongest-guide rate, guide choice does not matter and the project stops at that measurement, which is itself a useful negative result.

Why not just pick the guide by embedding similarity of traces, or from the same model family? These are cheap proxies for fit and could plausibly capture most of it. We include both as baselines. The collaborative predictor earns its place only if observed rescue outcomes on theorem-sharing neighbors carry information beyond surface similarity. TheoremQA's finding that definitions alone fail suggests that the useful signal lies in how a theorem is applied, which text similarity may not see. The experiment decides it.

Why not just train the receiver to use hints, as CoRe does? Training is complementary, not competing. It changes one pool of models and activates only when a peer succeeds within that pool. Our selector can sit on top of a CoRe-trained pair. The look-alike-trace arm is aimed specifically at the both-fail cases CoRe cannot reach.

Why not just ask the receiver, or a judge model, to choose among several hints? Thinking in a Crowd shows that reasoning models do not reliably reject misleading context, and reading several candidate hints multiplies exposure to it. We run this as a baseline with all candidate hints in context. We expect it to lose on harm rate even if it gains some rescues.

Proposed Experiments

The decisive first check is cheap. Build the rescue log for TheoremQA and MATH-500 and compare the hindsight best-guide rescue rate against the strongest-guide rate. A large gap justifies everything that follows.

The main experiment reports, on held-out problems, the rescue rate at a budget of one hint, NDCG over guides, the harm rate on problems the receiver already solves, and extra tokens spent. It covers TheoremQA (where GPT-4 with program-of-thought reaches 51.0% and open models stay below 15%), MATH-500, GPQA and MedMCQA. Results are broken down by theorem domain. For GPQA and MedMCQA, no reported guidability result was found on Lacuna, so our rerun of the strongest-guide protocol defines the reference.

The cold-start check asks how few calibration problems are needed to place a new receiver model, varying the count from 10 to 100, in the spirit of taste-matching. The both-fail check measures how often the look-alike-theorem traces rescue receivers on TheoremQA problems that every guide failed, against a baseline of nearest-embedding solved problems used as demonstrations.

A clear negative outcome would be any of three results: the oracle gap is small, the feature classifier with no neighborhood structure matches the collaborative predictor, or abstention fails to bring harm below that of the strongest-guide baseline. Each would tell us where the guidability ceiling actually comes from.