Reasoning language models are increasingly served with adaptive compute. A small learned predictor estimates how much each query would gain from more thinking, and the system gives hard-but-solvable queries more samples or reasoning tokens while giving easy or hopeless ones less. In a real cluster, the natural next step is to make these budgets react to load: think less when queues are long, more when the GPUs are idle. This proposal argues that this step recreates a well-known networking failure called route flapping. Route flapping happens when traffic is steered by a cost that itself depends on the traffic, so the system keeps swinging back and forth instead of settling. Here, cutting budgets drains the queue. The drained queue then invites larger budgets, and the queue fills again. Because a reasoning request's completion signal arrives only after thousands of tokens, the controller always acts on stale information, which is the classic recipe for this kind of oscillation.
We propose to control load-aware budgets with a single slowly updated "price of compute." The novel part is how fast that price is allowed to move. We set its update speed from the difficulty predictor's own output: the predictor tells us how strongly the whole population of pending queries will change its demand when the price moves. Combining that sensitivity with the measured feedback delay gives an explicit stability condition. The claim is that an allocator tuned this way keeps the accuracy benefits of adaptive compute and avoids the budget and latency swings that load-reactive allocators produce. It should also keep working across traffic patterns and difficulty mixes where any fixed damping constant tuned on one trace fails.
What prior work establishes and leaves open
The adaptive-compute side supplies the method. Learning How Hard to Think trains a marginal-reward predictor on hidden states that estimates the expected accuracy gain from more Best-of-k samples. It reaches the same success rate as uniform Best-of-k on math and code with half the samples, and up to 10% higher success at a fixed budget. It also reports that continuous online optimization over a batch behaves erratically under prediction noise, while discrete difficulty bins act as a regularizer. We reuse the predictor and take that erratic behavior as a first sign of instability. That paper allocates within static batches and never models arrivals, queues, or delay. Hidden-state probes such as The LLM Already Knows and Optimizing Reasoning Efficiency through Prompt Difficulty Prediction show that difficulty can be read before any token is generated. We need exactly that, because the price controller must know each query's demand curve at admission. Neither probe looks at backend load. The file notes that an all-hard burst would send everything to one tier.
The serving side supplies the setting. CascadeServe switches between precomputed configurations ("gears") by request rate and has no hysteresis, so it can chatter at gear boundaries. Cascadia jointly optimizes GPU deployment and escalation thresholds, and it recognizes that thresholds change downstream arrival rates. But it re-plans from offline latency tables and does not analyze the resulting feedback loop. Both route among model tiers. Neither controls per-query test-time budgets.
The networking side supplies both the diagnosis and the constraints. Transaction-routing "swapping" in distributed databases (Yu et al.) and circular spanning-tree recomputation in software-defined networks (A Dynamic Method of Constructing MST Based on OpenFlow Protocol) are the same loop in other systems. The constraining results matter most. Das et al. found that mandatory hold times often fail when load-dependent costs swing widely. Mitigating network noise on Dragonfly networks found oscillation whenever congestion shifts faster than the controller samples. Load Balancing in Compute Clusters with Delayed Feedback (POL) shows that greedy queue-based routing collapses under delayed acknowledgments, and it fixes this with particle-filter belief tracking plus tree search. That fix works, but it is expensive per request, and it treats jobs as opaque with fixed sizes. In our setting, the size of each job is the decision being made.
Core Claim
Each arriving query $i$ receives the budget $b_i^*(\lambda)=\arg\max_b\,[\hat\Delta_i(b)-\lambda\,c(b)]$. Here $\hat\Delta_i(b)$ is the predicted accuracy gain at budget $b$, $c(b)$ is its GPU cost, and $\lambda$ is a shared price. The price is raised when smoothed queue backlog exceeds a target and lowered when it falls below. Treat the cluster's response as a delayed integrator. Let $\kappa$ be the price update gain, $\varepsilon=-\partial(\text{offered load})/\partial\lambda$ be the demand elasticity, and $\tau$ be the feedback delay. The loop's crossover frequency is then $\kappa\varepsilon$. Requiring positive phase margin gives
$$\kappa\,\varepsilon\,\tau<\frac{\pi}{2}.$$
Flapping follows directly from this condition. The elasticity $\varepsilon$ is not constant. It is large when many pending queries sit near the point of indifference between two budgets, and near zero when they are dominated by "impossible" queries that already receive the minimum. The delay $\tau$ grows with the budgets themselves. So any fixed gain is either sluggish or unstable, depending on the traffic mix. Because $\hat\Delta_i$ is known at admission, we can compute $\hat\varepsilon$ online as the summed budget change of pending queries under a small price perturbation. We then set $\kappa=\alpha\pi/(2\hat\varepsilon\hat\tau)$ with $\alpha<1$. This gain scheduling is the key ingredient. It needs no search or particle filter, and it costs one extra predictor evaluation per queued query per control step.
Research Program
The work has three parts. The first is a measurement study. We add a simple load-reactive budget rule to an existing adaptive allocator and replay bursty traces through a real serving stack. The goal is to document flapping: periodic swings in mean budget, p99 latency, and success rate at a frequency set by the reasoning-length delay. This establishes that the problem is real for test-time budgets, which the file shows no prior work measuring.
The second is the controller. It uses the price rule above, estimates $\hat\varepsilon$ from predictor outputs over the pending set, estimates $\hat\tau$ from recent completion times, and schedules the gain. We will analyze how discrete budgets (integer k, or token quotas in steps) make $\varepsilon$ spiky. That spikiness is the mechanism behind boundary chattering in both binned allocation and CascadeServe gears. The analysis will show when a small randomization of $b^*$ near indifference smooths the demand curve.
The third is a robustness analysis under predictor error. Overconfident $\hat\Delta$ on impossible queries inflates both $\hat\varepsilon$ and real load. We will test whether a calibration step in the style of Thought calibration is needed to keep the stability margin honest.
Baselines
The naive reference point is uniform compute: a fixed Best-of-k (or fixed thinking quota) for every query, served with join-the-shortest-queue. We will set k to the largest value that meets the latency service-level objective at median load. This is what most deployments run, and it defines the floor that adaptive methods must clear on accuracy per GPU-second.
The strongest adaptive competitor is the marginal-reward allocator of Learning How Hard to Think, used in its offline binned form, which that paper found more reliable than online optimization. It is load-blind, so its bins are tuned to meet the objective at the traces' peak load. We will use the same predictor, trained on the same data, as in our method, so that only the control policy differs. Its reported result is our reference: equal success to uniform Best-of-k with half the samples on GSM8K, Numina-CoT and TACO, and up to 10% higher success at fixed budget. To beat it, our controller must reach higher success at equal SLO attainment on bursty traces, or equal success with measurably higher SLO attainment.
For the tier-routing view we include a Cascadia-style periodic re-optimizer. It re-solves thresholds every T seconds from recent statistics, with T tuned by grid search on a held-out trace. Cascadia reports 20–39% lower cost than CascadeServe on MT-Bench, and 2.3× tighter latency objectives on average. We found no reported result on Lacuna for any of these systems on per-query reasoning budgets or on oscillation metrics.
We also run a load-reactive naive allocator (price from instantaneous queue) and a hysteresis/hold-time variant, each tuned with the same grid budget as our controller. The key ablation is our price controller with a fixed gain κ tuned on one trace, compared against the same controller with the gain scheduled from $\hat\varepsilon$. This isolates whether predictor-derived elasticity matters. Every method gets the same GPUs, traces, and hyperparameter search budget.
Why not just…?
Why not just keep uniform compute and a standard load balancer? It cannot flap, which is its virtue. But the evidence in Learning How Hard to Think shows uniform Best-of-k needs twice the samples for the same accuracy. Stop Overthinking shows uniform long reasoning can even lower accuracy. POL's results show that shortest-queue routing degrades under delayed feedback anyway. Our experiments report accuracy per GPU-second against this baseline, so the cost of stability without adaptivity is measured directly.
Why not just add hysteresis or a hold time? Das et al. report that hold times often fail when load-dependent costs swing widely. Our stability condition predicts why: a hold time adds delay τ without reducing loop gain. This is a genuine threat, though. A well-tuned smoothing band could suffice on a single trace. We tune it with the same budget and test it on traces it was not tuned on.
Why not just use a slow fixed-gain integral controller? This is the most serious alternative, and the ablation exists to settle it. If a single κ tuned on the Azure trace also holds up on the Twitter trace and under shifted difficulty mixes (for example, a surge of AIME-level problems), then gain scheduling adds nothing. We would then report the simpler controller as the result.
Why not just use POL-style belief tracking? It handles delay well. However, it runs a tree search per request and assumes jobs of fixed size, whereas here the size of each job is the decision variable. We will run it with job sizes set by the binned allocator. This tests whether better state estimation alone removes flapping.
Proposed Experiments
We will serve a distilled reasoning model with Best-of-k and a capped thinking budget under a vLLM-style server. The queries are drawn from GSM8K, MATH-500, and TACO, and arrivals are replayed from the Azure Functions and Twitter traces used by CascadeServe. We will add a difficulty-shift condition that injects bursts of hard problems in the style of AIME24. Metrics are verified success rate, SLO attainment, p99 latency, and GPU-seconds per solved query. Oscillation is measured by the variance of mean budget and the spectral power of the latency series near the predicted frequency $\kappa\varepsilon$.
The decisive checks come in three steps. First, the measurement study must show that load-reactive allocation actually flaps, with a period that tracks reasoning length. If it does not, the premise fails. Second, when the gain is deliberately swept, the onset of oscillation should occur near $\kappa\hat\varepsilon\hat\tau\approx\pi/2$. This tests whether the stability condition, and hence the predictor-derived elasticity, is predictive. Third, on held-out traces and under difficulty shift, the gain-scheduled controller should match or beat the binned allocator on success at equal SLO attainment, while the tuned fixed-gain and hysteresis variants lose stability or accuracy. If the fixed-gain variant keeps pace, we will say so plainly.
References
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Optimizing Reasoning Efficiency through Prompt Difficulty Prediction
- CascadeServe: Unlocking Model Cascades for Inference Serving
- Cascadia: An Efficient Cascade Serving System for Large Language Models
- Robust Transaction-Routing Strategies in Distributed Database Systems
- A Dynamic Method of Constructing MST Based on OpenFlow Protocol
- Context-based Routing: Technique, Applications, and Experience
- Mitigating Network Noise on Dragonfly Networks through Application-Aware Routing
- Load Balancing in Compute Clusters With Delayed Feedback
- Thought calibration: Efficient and confident test-time scaling
- Dynamic Early Exit in Reasoning Models