Input Adaptive Compute Allocation for Language Models

1,628 concepts1,253 papers35 proposals

When a human is asked "What is the capital of France?", they answer instantly. When asked to prove a complex mathematical theorem, they pause, sketch out ideas, and deliberate.

Standard Large Language Models (LLMs), however, treat both queries with the exact same structural weight. In a standard autoregressive forward pass, every token costs the same amount of computation, regardless of whether it is a trivial punctuation mark or a critical logical turning point. As highlighted in A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems by Zixuan Ke et al., this uniform application of computational resources introduces severe inefficiencies: simple tasks are over-processed, while complex reasoning tasks are artificially constrained by fixed inference budgets.

To overcome this, a rapidly growing body of research focuses on input-adaptive compute allocation—developing mechanisms that allow models to dynamically scale their "thinking time" or computational depth based on the difficulty of the problem at hand.

The Cost of Uniform Compute

Why does uniform compute allocation fail? Consider test-time scaling methods like Chain-of-Thought (CoT) or Self-Consistency (SC), where a system samples multiple reasoning paths and takes a majority vote. If a system samples a fixed budget of 100 paths for every single question, it incurs massive financial and latency costs on easy questions where the first path would have sufficed. Conversely, for a highly complex problem, 100 paths might be insufficient to find a correct consensus.

As Mehul Damani et al. observe in Learning How Hard to Think: Input-Adaptive Allocation of LM Computation, real-world deployments process batches of queries with highly heterogeneous difficulty profiles. Without a mechanism to dynamically shift compute from trivial or impossible tasks to hard-but-solvable ones, a global computational budget cannot be optimized.

Uniform Compute:
[Easy Query]  --> [ Layer 1 -> Layer 2 -> ... -> Layer N ] --> (Wasteful Overthinking)
[Hard Query]  --> [ Layer 1 -> Layer 2 -> ... -> Layer N ] --> (Insufficient Depth)

Adaptive Compute:
[Easy Query]  --> [ Layer 1 -> Early Exit ]                 --> (Resource Saved)
[Hard Query]  --> [ Layer 1 -> Layer 2 -> ... -> Deep Search ] --> (Success)

How Models Sense Difficulty

To allocate compute adaptively, a system must first determine how hard a problem is before wasting resources on it. Researchers have uncovered two primary ways to achieve this: internal probing and routing.

Hidden Representations

Can an LLM gauge difficulty before generating a full response? In The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations, Yubo Zhu et al. demonstrate that an LLM's internal hidden states at the very beginning of inference contain sufficient information to predict whether the model will struggle with the prompt. By analyzing these early representations using a value function, systems can flag complex queries before outputting a single token.

Routing and Probing

Similarly, Bo Zhao et al. in Optimizing Reasoning Efficiency through Prompt Difficulty Prediction use intermediate representations to route easy questions to smaller, cheaper models while reserving larger, reasoning-heavy models for complex bottlenecks. This difficulty-aware routing matches the accuracy of top-tier models at a fraction of the computational cost.

Mechanisms for Dynamic Inference Scaling

Once difficulty is estimated, the system must dynamically adjust its resource expenditure. Several paradigms have emerged to handle this at inference time.

Early Stopping and Calibration

Instead of generating a fixed number of tokens or sampling paths, systems can dynamically halt when confidence thresholds are met. For instance, Yiwei Li et al. introduce Early-Stopping Self-Consistency (ESC) in Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning, which terminates sampling as soon as a statistically confident consensus is reached within a small window.

To ensure these stopping decisions are reliable, Menghua Wu et al. propose Thought calibration: Efficient and confident test-time scaling. They use lightweight probes to statistically guarantee performance while slashing token usage by up to 60%.

Dynamic Token Quotas

Rather than adjusting the number of parallel samples, frameworks can control the length or depth of the reasoning chain itself. In DRQA: Dynamic Reasoning Quota Allocation for Controlling Overthinking in Reasoning Large Language Models, Kaiwen Yan et al. use reinforcement learning to train models to internalize resource efficiency, dynamically shrinking token generation for simple steps and expanding it for complex bottlenecks.

Zhengkai Lin et al. similarly explore this in Controlling Thinking Speed in Reasoning Models, adjusting the model's "thinking speed" at inference time without retraining by modifying token generation boundaries based on real-time difficulty.

Adaptive Search and Planning

For multi-step reasoning tasks that rely on explicit search algorithms (like Monte Carlo Tree Search), uniform allocation is particularly damaging because it treats every branch of a decision tree with equal weight.

Pointwise Mutual Information

Jiaxi Li et al. develop MITS in MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information, which scores reasoning paths using Pointwise Mutual Information (PMI). By rewarding steps specifically relevant to the question and penalizing generic "filler" logic, it avoids over-computing simple steps and focuses exploration on high-uncertainty bottlenecks.

Balancing Fast and Slow Planning

In System 1.x: Learning to Balance Fast and Slow Planning with Language Models, Peter Hase demonstrates that models can be trained to dynamically alternate between fast, single-pass generation ("System 1") and slow, deliberative tree search ("System 2") depending on the local difficulty of individual sub-goals within a task. This prevents the model from wasting time verbalizing trivial steps while ensuring deep exploration where it matters.

Extending to Training Dynamics

The inefficiency of uniform compute isn't confined to inference; it also plagues reinforcement learning (RL) workflows. When fine-tuning reasoning models, traditional pipelines assign a fixed number of rollouts or attempts to every training prompt.

In Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation, Ziniu Li et al. frame training budget allocation as a classical knapsack problem. They adaptively funnel exploration compute toward "teachable" tasks while avoiding waste on prompts that are either trivial or completely unresolvable at the model's current state.

Jiarui Yao et al. echo this principle in Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL, showing that dynamically allocating sampling budgets to prompts with the clearest learning signals minimizes gradient variance and accelerates training efficiency by 2-4x.

By treating compute as a dynamic, fluid resource rather than a static constraint, input-adaptive allocation bridges the gap between raw parameter scaling and efficient, human-like cognitive flexibility.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_uniform_computational_allocation_uniform_eaf61539,
  title = {uniform computational allocation (uniform compute allocation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uniform-computational-allocation-uniform-compute-allocation/rdr_eaf61539af744c16b8617c9e62fb567b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_uniform_computational_allocation_uniform_eaf61539_v1,
  title = {uniform computational allocation (uniform compute allocation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uniform-computational-allocation-uniform-compute-allocation/rdr_eaf61539af744c16b8617c9e62fb567b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake