Robot policies learned from human demonstrations inherit one fixed trade-off between speed, effort, and reliability: whatever the demonstrator happened to do. That trade-off is a hidden static-price assumption. It implicitly treats time and energy as having constant, unchanging costs. In deployments where electricity tariffs swing several-fold within a day, or where throughput deadlines tighten and loosen, the right behavior changes with the price. A cloned policy cannot follow, because it never sees cost at all. We propose to keep a large imitation-learned policy frozen as a generator of candidate motions. On top of it we train a small critic, a network that predicts future outcomes of each candidate. The critic has separate output heads for task success, elapsed time, and energy use per future time bin. At run time the robot scores candidates by combining these heads with the current price forecast. When prices change, it re-ranks its own candidates without any retraining.
The decomposition matters for transfer from simulation to hardware. Of the three quantities, prices have no sim-to-real gap, because they are observed exactly. Success has the familiar gap that domain randomization targets. Energy has a different gap: simulated motor power is often wrong, but real motor power can be logged directly on hardware without any task reward. Splitting the critic along these lines localizes the reality gap to one head that can be calibrated cheaply. The claim is that this gives zero-shot price adaptation and near-zero-shot transfer of cost-aware behavior. Neither a cloned policy nor a single success-only critic can provide both.
Core Claim
Let $\lambda_{1:K}$ be a forecast of the energy price over the next $K$ time bins and $\mu$ the current value of time. For a state $s$ and candidate action chunk $a$ drawn from the frozen policy, the critic outputs three things: $Q_S(s,a)$, the probability of eventual success; $Q_T(s,a)$, the expected remaining time; and $E_k(s,a)$, the expected energy consumed in future bin $k$. The robot executes the chunk maximizing
$$ J(s,a) = R\,Q_S(s,a) - \mu\,Q_T(s,a) - \sum_{k=1}^{K} \lambda_k\,E_k(s,a), $$
where $R$ is the value of completing the task. Because cost is linear in price, this objective is exact for any price path the forecast describes. That is why one critic can serve every price regime.
We make three predictions. First, across randomized price traces, this critic dominates both the cloned policy and a success-only critic on the success/time/energy trade-off. Second, it approaches an oracle critic retrained for each fixed price. Third, after calibrating only $E_k$ on logged real motor power, its energy predictions and cost-adjusted returns on hardware match simulation far more closely than those of a critic whose single scalar output mixes all three quantities.
Relation to Prior Work
The mechanism we reuse comes from Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. That work freezes a large cloned policy, samples $N$ action chunks from it, and executes a softmax-weighted average scored by a roughly 1B-parameter critic trained with categorical value regression on binary success rewards. It breaks the imitation ceiling on success but has no notion of cost. Its critic would need full retraining each time the trade-off changed. We keep its architecture and training recipe and change what the critic predicts and how its outputs are combined.
Two papers motivate the claim that imitation is blind to cost. CADO shows that in neural combinatorial solvers, similarity to expert labels barely predicts solution cost ($R^2 = 0.076$), and fixes this with reinforcement-learning fine-tuning against true cost. Its costs are static graph weights, whereas ours move with an external signal. APC-RL exceeds suboptimal demonstration priors by composing prior-based actors with an unconstrained actor under a Q-value selector. It is a competing route past the ceiling, but it requires training new actors and has no physical transfer.
The static-pricing literature supplies the motivation that fixed-cost optimization breaks under volatile prices. To Reserve or Not to Reserve makes this point for cloud spot markets, the park microgrid study for storage-versus-export decisions, and The Orchestrator Bottleneck for agent workflows. None of these involve embodied control or learned policies.
On transfer, Sim-to-Real: Learning Agile Locomotion For Quadruped Robots shows two things relevant to us. Energy efficiency of learned gaits can exceed expert tuning, by 35% for galloping. Doing so required custom motor models and measured latencies. This tells us energy is exactly the quantity where naive simulation is untrustworthy. Asymmetric Actor Critic gives the critic information the actor lacks. We follow that pattern by giving only the critic the price forecast, while the frozen policy never sees prices. Reinforced Grounded Action Transformation grounds simulators with real transitions. Our energy calibration is lighter: it fits only one critic head to power measurements and leaves the dynamics untouched. RoboTwin 2.0 supplies the strongly randomized simulation used to train the success head.
Research Program
We first instrument simulation for energy. In RoboTwin 2.0 and LIBERO (whose underlying simulator, Robosuite, exposes joint torques), we log per-step electrical power, estimated as torque times velocity plus a randomized motor-loss term. The randomized loss term covers copper loss, idle holding current, and gripper actuation. Episodes also sample price traces from mean-reverting processes with spikes. Historical day-ahead tariff traces will be added if a suitable public source is chosen; none is indexed in the file.
The frozen base policy is a diffusion-style cloned policy of the kind studied in Diffusion Policy, trained on standard demonstrations. Because candidate support limits what the critic can promote, we widen the candidate set cheaply. Besides ordinary samples, it includes time-rescaled copies of each chunk at several speed factors, plus samples at higher sampling temperature. This keeps the base policy frozen while giving the critic slow, low-effort and fast, high-effort variants to choose between.
The critic is trained off-policy on simulated rollouts. Success uses categorical value regression as in Q-Planning. The time and energy-per-bin heads use the same distributional regression, with Monte-Carlo or temporal-difference targets.
For hardware, we roll out the frozen policy plus the simulated critic on a dual-arm platform of the RoboTwin type, logging motor currents and voltages. We then refit only the energy heads on these logs. Because power is measured whether or not a task succeeds, this calibration needs no reward labels and no human resets beyond normal operation. The success head optionally continues to improve from deployment outcomes, as in Q-Planning.
Baselines
The naive reference is the frozen cloned policy executed as-is, which represents the static-price status quo. Its success, time, and energy are fixed regardless of price. Any method that cannot beat it on cost-adjusted return under price swings is pointless.
The strongest existing method is success-only Q-Planning from Beyond Imitation. We will run it with the same frozen policy, the same widened candidate set, the same critic size, the same rollout budget, and the same hyperparameter search budget as our method. Its reported numbers are 83.8% to 91.4% success on the 47-task RoboTwin suite and 93% to 99% on LIBERO-10. On hardware it reached 90% on stack-cups and 80% on insert-wallet, while filtered fine-tuning stalled at 55% and 30%. No reported result was found on Lacuna for energy use or cost per successful episode under any of these methods, so the cost comparison is new measurement rather than reproduction. Beating it means a higher mean cost-adjusted return $J$ across held-out price traces. It also means success within two points of it, because buying energy savings with lost reliability would not count.
A second competitor is a price-indexed oracle: a separate success-time-energy scalar critic trained for each of a grid of fixed price levels, then selected by the current price. This represents re-solving a static problem whenever prices change. We give it the same total rollout budget split across the grid. Our goal is to match it within a small regret while using one model. We also want to exceed it under within-episode price changes, which it cannot represent.
The key ablation is a monolithic critic. It receives the price forecast as an input and outputs a single scalar return, using the same data and capacity as our method. This isolates the decomposition: if it matches our method in simulation and after hardware calibration, the separate heads add nothing.
Finally, a speed-knob heuristic chooses the time-rescaling factor of the cloned policy's chunks from the current price via a tuned lookup table, with no critic.
Why not just…?
Why not just collect demonstrations at different prices and give the price to the cloned policy? Human teleoperators cannot perceive motor energy, so price-conditioned demonstrations would mostly encode a vague "go slower" instruction. Such data would also hit the same ceiling documented for filtered fine-tuning in Q-Planning (55% and 30% on hardware). CADO's $R^2 = 0.076$ result suggests that matching demonstrations is a poor proxy for cost. We will include a price-conditioned cloned policy trained on simulated scripted demonstrations at three effort levels. That gives it the most favorable possible data.
Why not just scale speed with price? This is the real threat. If energy trade-offs in these tasks reduce to moving slower, the speed-knob heuristic will capture most of the gain, and the critic is overkill. Our hypothesis is that energy also depends on which candidate is taken: a grasp that avoids holding a payload against gravity, fewer regrasps, shorter idle holding. Speed scaling also raises failure risk on contact-rich tasks, which only a success-aware scorer can weigh. Because time-rescaled chunks are in our candidate set, the comparison is clean. If the critic's chosen candidates rarely differ from the best rescaled chunk, or the heuristic comes within noise of our cost-adjusted return, we will report that the phenomenon is speed-only and the contribution shrinks to the calibration result.
Why not just widen domain randomization over the energy model instead of calibrating? Wide randomization makes policies optimize against worst cases, which the domain-randomization literature associates with conservative, slow behavior. For a cost critic, this would systematically overestimate energy and over-favor slow candidates. We test this directly by comparing a critic trained under a much wider motor-loss range against the narrow-range critic after head calibration, on real energy prediction error and real cost-adjusted return.
Why not just retrain the critic whenever prices change? That is the price-indexed oracle. It is affordable for a few static price levels. It cannot handle prices that change within an episode, and it multiplies training cost by the grid size. The baseline measures exactly how much is lost by not doing it.
Proposed Experiments
In simulation, we train on RoboTwin 2.0 and LIBERO-10 with randomized physics, visuals, motor losses, and price traces, and evaluate on held-out price traces. These include out-of-range spikes and mid-episode regime switches. Metrics are success rate, mean time, mean energy, mean cost-adjusted return $J$, regret against the price-indexed oracle, and the area of the success/time/energy trade-off front traced by sweeping $\mu$ and $\lambda$.
On hardware, we use two contact-rich bimanual tasks in the spirit of stack-cups and insert-wallet, with logged motor power. We report three things for each condition (simulated critic as-is, after energy-head calibration, and the monolithic critic after equal calibration effort): energy prediction error, the gap between simulated and real cost-adjusted return, and success.
The decisive checks are:
- Whether decomposed heads beat the monolithic critic under mid-episode price switches. This is where linear price combination should matter most.
- Whether calibrating the energy head alone closes most of the real-versus-simulated cost gap while leaving success unchanged.
- Whether the critic chooses non-rescaled candidates often enough, and profitably enough, to beat the speed-knob heuristic.
If all three hold, a frozen cloned policy can be re-priced at deployment. If the third fails, we will say so plainly.
References
- Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- CADO: From Imitation to Cost Minimization for Heatmap-based Solvers in Combinatorial Optimization
- APC-RL: Exceeding data-driven behavior priors with adaptive policy composition
- To Reserve or Not to Reserve: Optimal Online Multi-Instance Acquisition in IaaS Clouds
- Research on the Optimization of Park Microgrids Based on Renewable Energy Utilization and Storage Systems
- The Orchestrator Bottleneck: Formal Coordination Strategies for Cost-Optimal Multi-Agent Enterprise Workflows
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- Asymmetric Actor Critic for Image-Based Robot Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion