When a reasoning model gets a batch of questions and one shared token budget, it spends that budget badly. Thinking Hard, Not Smart documents this directly. Models solve questions in the order they appear: the correlation between solving order and position averages $+0.68$ and reaches $+0.99$. They lose 16 to 19 points when hard, low-value questions come first, and at $N=20$ some models attempt only about 23% of the questions. The obvious fix is to take the budget decision away from the model and give it to an external allocator. That fix already exists in another form. Learning How Hard to Think trains a small predictor of how much each extra unit of compute will help a query, then splits a global budget across a batch. Bolting that allocator onto the exam setting would probably recover much of the lost score, and it would be an engineering note rather than a research contribution.
This proposal targets what that allocator, and every allocator like it, assumes away: that the queries are independent. Database systems learned long ago that the best plan for a whole workload can include a plan that is worse for one query, because its intermediate results make other queries cheaper (Algorithms for Materialized View Design). Reasoning batches have the same structure. A lemma, a closed form, or a traced helper function derived for one problem can make a related problem cheap. We propose an allocator that counts this spillover: when it decides how much to spend on a query, it credits both that query's own expected gain and the expected gain its reusable by-products give to other queries still waiting. A hard, low-value problem may then be worth solving early if it unlocks several high-value ones.
Core Claim
Let a batch contain queries $i=1..N$ with point values $v_i$ and a shared budget $B$. Let $c_i$ be the tokens spent on query $i$, and let $p_i(c_i \mid A)$ be its success probability given the set $A$ of artifacts (verified-looking intermediate results) produced so far. Independent allocators maximize $\sum_i v_i\, p_i(c_i)$ subject to $\sum_i c_i \le B$, with $A$ ignored. We instead rank the next unit of compute with the marginal value
$$ g_i = v_i\,\Delta p_i \;+\; \sum_{j \text{ unsolved}} v_j\, \Delta p_j(\text{artifacts of } i), $$
where the first term is query $i$'s own predicted gain from more compute and the second term is the predicted gain it passes to every other unsolved query $j$. The claim is narrow and testable. On batches whose questions share sub-derivations, a greedy allocator that ranks by $g_i$ beats the best independent allocator that has the same artifact-sharing mechanism. The gain should grow with measured overlap in the batch. On batches with no overlap, the gain should vanish without becoming negative.
The second term is what is new. Experience sharing alone is not new: Do Not Waste Your Rollouts recycles "truth anchors" and "dead ends" across rollouts of a single problem. Per-query cost prediction is not new either. Allocators that treat queries as independent items in a knapsack, a budget-packing problem, already exist in Learning How Hard to Think for inference and in Knapsack RL for training. What is missing is cross-query sharing whose value enters the budget decision.
Research Program
The system has three parts.
Artifact store. After each attempt on a query, the model writes out short reusable statements: intermediate identities, solved sub-cases, or the traced behavior of a code fragment. These go into a deduplicated store, much as RSE builds its experience bank. Every token spent writing an artifact or reading it back is charged to the budget, so savings are net.
Spillover predictor. This is a pairwise model that estimates $\Delta p_j$ when query $i$'s artifacts are put into $j$'s context. We train it offline on held-out workloads by direct intervention. We solve $i$, inject its artifacts into $j$, and measure the change in $j$'s success rate over repeated samples. Inputs are prompt-level hidden states of both queries, following the evidence that pre-generation states encode difficulty (The LLM Already Knows), plus a similarity score between the artifact text and query $j$. Pairwise prediction costs $O(N^2)$, but each call is a small multilayer perceptron (a two-layer neural network), negligible next to generation at $N \le 50$.
Scheduler. The scheduler greedily allocates rollout chunks by $g_i$ and re-estimates after each chunk as artifacts accumulate. It borrows the "build a sharing graph, then pick a plan" structure of BatchEnum and Collective Grounding. Both show that deciding which shared intermediate results to materialize is where workload-level savings come from. Collective Grounding reports up to a 70% runtime reduction, and BatchEnum up to a 7.7× speedup. We do not inherit their exact containment checks, because reasoning overlap is soft and probabilistic. That is why spillover is predicted rather than computed.
Safety mechanism. Sharing has a failure mode that does not exist in databases: a wrong lemma can spread. ESC already fails when a model is confidently wrong on one query (Escape Sky-high Cost). Cross-query sharing could turn that into correlated errors across a batch. To limit this, each artifact is tagged with agreement statistics across rollouts, and only artifacts above a consistency threshold are injected. We measure error propagation explicitly rather than assume it away.
Baselines
Naive reference points. The do-nothing baseline is Fan et al.'s setting: the whole exam in one prompt with budget $B$, plus their explicit-planning prompt as the strongest in-context variant. The second reference point is a uniform per-query split, $B/N$ tokens each, run in isolation. Every method uses the same models as Fan et al. (Qwen3 8B and 14B, and the DeepSeek-R1-Distill models), the same point values, and the same total token accounting.
Strongest independent allocator. This is the marginal-reward predictor of Learning How Hard to Think, retrained on our training split with the same tuning budget we give our predictor. It reports matching best-of-$k$ success on GSM8K with half the samples, and up to 10% quality gains at fixed budget on math and code. We found no reported result on Lacuna for it under the shared-budget exam setting on Omni-MATH, so we establish that number ourselves. We also run per-query ESC and thought calibration inside the independent allocator's per-query budgets, so local early stopping is not a missing ingredient.
Strongest sharing baseline. This is RSE extended to the whole batch: one experience bank shared by all queries, scheduled by the independent allocator. RSE reports 83.9% Pass@1 on HMMT25 with Qwen3-30B on single problems (Do Not Waste Your Rollouts). This baseline holds the sharing mechanism fixed and differs from our method only in the scheduler.
Key ablation. Our method versus the batch-RSE baseline isolates the spillover term in $g_i$. Our method versus the independent allocator with no store isolates sharing as a whole.
What counts as winning. Beating the independent allocator means a higher expected exam score at equal net tokens, with confidence intervals over batch draws that do not overlap. Beating batch-RSE requires a gain that rises with overlap, because only then is the scheduler, not the store, doing the work.
Upper bound. An oracle allocator that knows the true $\Delta p$ and the true spillover values bounds what predicted spillover can achieve.
Why not just…?
Why not just prompt the model to plan its budget? Fan et al. tested exactly this. Explicit planning raised coverage but did not produce allocation that tracked difficulty or point value, and order bias persisted. We still include it as a baseline, because newer models may do better.
Why not just run an independent difficulty-aware allocator? This is the real threat on ordinary batches, and we expect it to close most of the order-bias gap. Our claim is only about batches with shared structure. The overlap-stratified experiments below test it: if spillover-aware scheduling shows no gain over the independent allocator at high overlap, the idea fails.
Why not just share everything and skip the allocator change? Putting all queries in one context with prefix caching (Compute or Load KV Cache?), or a batch-wide experience bank, captures reuse whenever it happens to occur. It does not change which query to spend on so that reuse happens. That simpler design is our main ablation. If it matches our method, we will report that the sharing mechanism, not the scheduling, is the contribution.
Why not just sort easy-to-hard? Allocation should follow how much extra compute helps each query, not raw difficulty, as both Learning How Hard to Think and Knapsack RL argue. Sorting also cannot express "solve the hard enabling problem first." We include easy-first ordering as a baseline.
Proposed Experiments
The primary testbed is Fan et al.'s exam protocol on Omni-MATH: $N \in \{5, 10, 20\}$, assigned point values, a shared budget $B$, and both easy-first and adversarial orderings. Batches come at three overlap levels:
- Random batches, where little sharing is expected.
- Topic-clustered batches, grouped by embedding similarity of problems and reference solutions.
- Constructed families, where problems share an explicit sub-result. Examples are a general identity followed by its specializations, or CRUXEval-O programs (CRUXEval) recomposed to call common helper functions whose traced behavior is reusable.
We first measure natural overlap, defined as the oracle spillover between pairs. If clustered real batches show almost none, the contribution shrinks to the constructed regime, and we will say so.
Metrics are value-weighted exam score at fixed net tokens, coverage, wall-clock time, and the propagation rate: the fraction of wrong answers traceable to an injected artifact.
Decisive checks. The first check is whether the gain of our method over batch-RSE increases monotonically with measured overlap and is roughly zero on random batches. The second is whether the spillover predictor's ranking correlates with oracle spillover on held-out families. The third is whether the propagation rate stays below the error rate of isolated solving. If any of these fails, the spillover term is not earning its cost.
References
- Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Algorithms for Materialized View Design in Data Warehousing Environment
- Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling
- Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Batch Hop-Constrained s-t Simple Path Query Processing in Large Graphs
- Collective Grounding: Applying Database Techniques to Grounding Templated Models
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning
- Thought calibration: Efficient and confident test-time scaling
- Compute Or Load KV Cache? Why Not Both?
- CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution