Refinement vs. Replanning: Inference Compute Allocation in Robot Policies

Alba Maria Tellez FernandezarXiv 2026 · arXiv

TL;DR

When deploying a robot policy, you face a critical trade-off: do you spend your compute budget on making each action prediction more precise (refinement), or do you call the model more often to react to new data (replanning)? This paper demonstrates that for the SmolVLA model, doing less refinement work per call actually improves task success while simultaneously reducing compute costs, challenging the intuition that more denoising steps always lead to better robotics.

What Problem Does This Paper Address?

Modern robot policies, particularly those based on Vision-Language-Action (VLA) models, often predict "chunks" of actions rather than a single step. At deployment, a researcher must decide how to allocate a fixed inference compute budget.

There are two main levers:

  1. Refinement ($d$): How much work (e.g., denoising steps in a diffusion-based model) should the model do to ensure the predicted action chunk is high-quality?
  2. Replanning Frequency ($h$): How many actions from a predicted chunk should the robot execute before it stops, looks at a fresh observation, and calls the model again?

While training compute is well-studied, the optimal allocation of inference compute for a frozen model is less understood. The authors investigate whether spending more on "thinking harder" (refinement) is better than "thinking more often" (replanning).

Methods

The authors propose a compute-accounting formulation to connect these two levers. They use a frozen SmolVLA checkpoint—a compact VLA model—and test it in a synchronous LIBERO simulation environment.

Key technical aspects include:

  • Control Variables: They fixed the total predicted chunk length ($K$) at 50 and varied $d$ (denoising steps: 2, 5, 10, 20) and $h$ (execution prefix: 1, 2, 5).
  • Two-Phase Evaluation: A "Discovery" phase on four spatial tasks was used to select the most promising configurations under specific compute ceilings (25%, 50%, and 100% of the native rate). These were then tested in a "Held-out" phase on eight different tasks to ensure the findings generalized.
  • Cost Metrics: They measured both nominal arithmetic (GFLOPs per control step) and realized attempt cost (total TFLOPs spent during an entire task attempt), noting that these don't always scale linearly with wall-clock latency.

Datasets

The study utilizes the LIBERO benchmark, a suite designed for lifelong robot learning.

  • Discovery Tasks: 4 "Spatial" tasks (e.g., picking and placing objects in specific spatial relations).
  • Held-out Tasks: 8 tasks categorized under "Goal" and "Object" suites (e.g., putting a cream in a basket or a bowl on a stove).
  • Scale: The final evaluation involved 1,120 planned attempts across these tasks to ensure statistical significance.

Key Insights and Ideas

  • The "Fixed Cost" Floor: A surprising 82.1% of the arithmetic cost in a SmolVLA call is independent of the number of denoising steps. This is due to the heavy lifting required for vision and language "prefill" (processing the image and text prompt). This means you can't save massive amounts of compute just by reducing denoising; the model has a high "entry fee" for every call.
  • Refinement is Not Monotonic: In robotics, "thinking harder" doesn't always help. The authors found that increasing denoising steps ($d$) sometimes decreased success. They hypothesize that more denoising steps might allow learned approximation errors to accumulate, pushing the robot's first few actions away from a successful trajectory.
  • The Sweet Spot: The configuration with $d=2$ (very low refinement) and $h=2$ (frequent replanning) emerged as a winner, outperforming the $d=5$ version by 10.6 percentage points while using less compute.
  • Arithmetic vs. Latency: The paper highlights that GFLOPs and host latency characterize cost differently. For instance, halving denoising steps saved only 8.9% in arithmetic but reduced measured host time by 40.5%. This suggests that "compute-optimal" settings depend heavily on whether your bottleneck is energy/FLOPs or real-time responsiveness.

Key Results

  • Success vs. Compute: In the primary comparison ($h=2$), moving from $d=5$ to $d=2$ raised the success rate from 76.3% to 86.9%.
  • Efficiency Gains: This success boost came with a 5.9% reduction in arithmetic per control step and a 15.6% reduction in total arithmetic per attempt.
  • Replanning Impact: Executing 5 actions ($h=5$) instead of 2 ($h=2$) at a low denoising level ($d=2$) yielded nearly identical success (86.3% vs 86.9%) but slashed the total arithmetic per attempt by 60.7%. This demonstrates that for some tasks, you can replan much less frequently without a significant performance penalty.

Limitations and Future Work

  • Model Specificity: The results are tied to one specific frozen SmolVLA checkpoint. The authors explicitly state they do not claim these exact ratios transfer to all other models.
  • Synchronous Constraints: The study uses synchronous simulation, which ignores "observation staleness"—the reality that in the real world, the world keeps moving while the robot is "thinking."
  • Fixed Chunk Length: The chunk length ($K=50$) was held constant. Future work could explore $K$ as a third lever in the compute allocation problem.
  • Feedback vs. Position: The authors note that changing the execution prefix ($h$) conflates two things: how fresh the observation is and which specific action in the chunk is being executed. Future studies should attempt to disentangle these effects.

Summary updated

Abstract

A frozen chunked robot policy can spend deployment-time inference compute in two coupled ways: more work within each policy call to refine a predicted action chunk, or more frequent policy calls to replan from fresh observations. We study this allocation without updating model weights. Our evaluation procedure reports success with both nominal compute rate and complete attempt cost, selects comparisons on discovery tasks, and tests prespecified comparisons on held-out tasks. Comparisons added after inspecting the held-out results only summarize observed patterns. We demonstrate the procedure with one frozen SmolVLA checkpoint in synchronous LIBERO simulation, using denoising count as the refinement control and fixing chunk length at K=50. Arithmetic and latency characterize cost differently: at d=10, 82.1% of counted arithmetic is independent of denoising, whereas a descriptive affine fit to fixed-input host time assigns 17.7% to its intercept. At h=2, d=2 succeeds on 139/160 attempts versus 122/160 for d=5 (+10.6 percentage points; task-conditional 95% interval: +4.4 to +16.9), while using 5.9% less arithmetic per control step. Executing h=5 rather than h=2 at d=2 yields 138/160 versus 139/160 successes and 60.7% lower arithmetic per attempt; equivalence is not established. The case study shows why deployment adaptation must evaluate generation effort and feedback frequency jointly; it does not define a universal setting.
BibTeX
@misc{fernandez2026refinement,
  author = {Alba Maria Tellez Fernandez},
  title = {{Refinement vs. Replanning: Inference Compute Allocation in Robot Policies}},
  year = {2026},
  eprint = {2611.19034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19034},
}
Figure