Reinforcement Learning with Computational Cost Penalties

108 concepts82 papers1 proposal

Standard reinforcement learning (RL) agents are typically optimized for a single objective: maximizing task performance. Whether playing a game, generating text, or controlling a robot, the agent is rewarded for success and penalized for failure. However, this objective ignores the "computational tax" required to achieve that success. In practice, this leads to "computational bloat," where models expend excessive resources—such as long reasoning chains, high-resolution image processing, or frequent replanning—on trivial tasks that could be solved more simply.

A growing body of research proposes a shift toward cost-aware reinforcement learning. By explicitly incorporating computational costs into the reward function, researchers are training agents that treat their own processing power as a finite, costly resource.

The Cost-Regularized Reward Function

The core mechanism behind this direction is a multi-objective reward function. Instead of rewarding only accuracy ($R_{task}$), the trainer introduces a penalty term proportional to the computational effort expended ($C$):

$$R_{total} = R_{task} - \lambda \cdot C$$

Here, $\lambda$ serves as a scaling factor that determines the trade-off between performance and efficiency. As Hu et al. argue, this approach mimics human cognition, where we instinctively reserve intense "System 2" deliberation for complex problems while relying on fast "System 1" intuition for simple ones.

The definition of "cost" varies by domain:

  • Large Language Models: Cost is typically measured in token usage or reasoning steps (Fang et al.; Jiang et al.).
  • Computer Vision: Cost may represent the number of pixels processed or the number of layers activated in a cascade (Gao et al.; Guan et al.).
  • Robotics and Control: Cost often refers to the frequency of model inferences or replanning cycles (Orenstein et al.).

Scaling Reasoning Effort in Language Models

In the context of Large Language Models (LLMs), cost-aware RL addresses the tendency of reasoning-heavy models to "overthink" simple queries. Fang et al. demonstrate that by penalizing excessive token usage, a model can learn to scale its reasoning depth dynamically, reducing inference costs by up to 90% without sacrificing accuracy.

A common strategy involves asymmetric reward structures. For example, Zeng et al. propose assigning higher rewards to successful outcomes achieved without internal reasoning compared to those achieved with it. This creates a training pressure that forces the model to prioritize efficiency, resorting to intensive deliberation only when a direct response is likely to fail. Similarly, Jiang et al. use an objective that samples both reasoning-heavy and direct paths, rewarding the shorter path if its performance meets a quality threshold.

To make these models more practical for deployment, He et al. introduced a two-phase strategy: first optimizing for maximum performance, then applying budget-aware reward shaping to allow users to toggle between "High," "Medium," and "Low" reasoning effort modes.

Adaptive Computation in Vision and Robotics

The principle of cost-awareness extends beyond text to spatial and temporal reasoning. In object detection, Gao et al. use RL to train a "pathfinder" that decides which regions of a high-resolution image require a "zoom-in" for finer analysis. By penalizing the number of processed pixels, the system learns to ignore background noise and focus compute on informative regions.

In robotics, agents often suffer from "computational rigidity," processing every frame with the same intensity. Orenstein et al. introduced agents that treat computation as a costly resource, learning to "think" during critical maneuvers and "coast" during predictable segments. This is echoed in the work of the authors of the Bernoulli-Continuation Policy, who use RL to decide when a robot should trigger a new planning cycle versus continuing an existing action, thereby reducing stale data failures and redundant inferences.

Training Dynamics and Convergence

Training cost-aware agents presents a unique challenge: if the cost penalty is too high too early, the model may never learn the complex reasoning required for difficult tasks. To solve this, Zhu et al. suggest a curriculum-based reward system. The model is initially incentivized solely for accuracy; once it masters the task, the efficiency penalty is gradually introduced.

This phased approach ensures the model first discovers the "correct" path before it begins pruning that path for efficiency. As Chen et al. note, standard RL objectives that ignore cost prevent models from learning frugal execution strategies. By integrating cost directly into the policy optimization loop, researchers are moving toward agents that are not just accurate, but operationally economical.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cost_aware_reinforcement_learning_07e28890,
  title = {cost-aware reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cost-aware-reinforcement-learning/rdr_07e288909e024da8b786e84863ad1d3e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cost_aware_reinforcement_learning_07e28890_v1,
  title = {cost-aware reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cost-aware-reinforcement-learning/rdr_07e288909e024da8b786e84863ad1d3e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake