ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan SongarXiv 2026 · arXiv

TL;DR

ExRL (Execution at the Right Length) is a sample-efficient method for adapting robot policies that predict long sequences of actions ("chunks"). Instead of trying to fine-tune high-dimensional action sequences, ExRL learns a simple scalar: how many steps of a predicted chunk to execute before replanning, enabling rapid real-world adaptation in as little as ten minutes.

What Problem Does This Paper Address?

Modern robot policies often use action chunking, where the model predicts a sequence of future actions (e.g., 50 timesteps) rather than just one. This helps capture complex temporal patterns and reduces the computational overhead of frequent model queries. However, these policies often struggle when deployed in new environments due to the "reality gap."

Adapting these models using standard Reinforcement Learning (RL) is notoriously difficult. Because an action chunk is high-dimensional (e.g., 50 steps across 7 robot joints = 350 dimensions), the search space for RL becomes enormous. This leads to poor sample efficiency, requiring hours or days of robot interaction time that is often impractical or unsafe in the real world.

Methods

The authors propose ExRL, which shifts the adaptation target from what actions to take to how long to execute the actions the policy has already predicted.

  1. Lightweight Adapter: ExRL adds a small neural network adapter ($\pi_{ex}$) that takes the current state and the predicted action chunk as input and outputs a discrete execution length $k$ (between 1 and the maximum chunk size $H$).
  2. Duration-Aware RL: The adapter is trained using off-policy RL (specifically ExRL-Q or ExRL-SAC). The authors formalize this as a Semi-Markov Decision Process (SMDP), where transitions have variable durations. The Bellman update is modified to account for the accumulated reward and the discount factor over the chosen execution length $k$.
  3. Online Distillation (Optional): While the base policy can remain frozen, ExRL can optionally use "Online Behavior Cloning" to distill successful adapted trajectories back into the base policy, allowing the underlying action generation to improve over time.
  4. Offline Data Reuse: ExRL can treat any sequence of primitive actions from human demonstrations as a "virtual chunk," allowing it to pre-train the adapter on offline data before the robot ever touches the real world.
Fig. 2. ExRL learns execution lengths with RL, and we optionally use Online BC to distill successful online rollouts back into the base policy during RL.

Datasets

The researchers evaluated ExRL across a wide variety of benchmarks and real-world tasks:

  • Simulation:
    • Robomimic: Standard manipulation tasks like Lift, Can, Square, Tool Hang, and Transport.
  • Adroit: High-frequency dexterous hand manipulation (e.g., Door, Pen, Hammer).
  • LIBERO: A benchmark for Vision-Language-Action (VLA) models.
  • Real-World: Four challenging tasks using an AgileX Piper arm: Cube Stacking, Clothing Folding, Power Insertion, and Parts Organization.
  • Base Policies: They tested ExRL on diverse architectures, including Diffusion Policy (DP), ACT (Transformers), and large-scale generalist models like $\pi_{0.5}$ and Cosmos Policy.

Key Insights and Ideas

  • Dimensionality is the Bottleneck: The core "aha!" moment is that the expressivity of a policy grows with chunk size, but the difficulty of adapting it via RL grows even faster. By optimizing a scalar (length) instead of a vector (actions), ExRL keeps the optimization space small regardless of how long the chunks are.
  • Adaptive Replanning vs. Fixed Horizons: Standard policies often use a fixed execution length. ExRL discovers that robots need different horizons for different phases of a task. For example, it learns to use long, open-loop motions during "free motion" (moving toward an object) but switches to short, frequent replanning during "near interaction" (precise grasping or insertion).
  • Overcoming Feature Aliasing: Sometimes, replanning too often is actually harmful. If the robot's sensors see very similar features in successive steps, the model might repeatedly predict the same "start" of a motion, getting stuck in a loop. ExRL learns when to "trust" the remainder of a chunk to break through these ambiguous states.
Comparison showing that while existing RL methods lose efficiency as chunk sizes grow, ExRL maintains high performance.

Key Results

  • Superior Sample Efficiency: In simulation (Robomimic), ExRL reached high success rates significantly faster than state-of-the-art baselines like DSRL or Policy Decorator. On the Tool Hang task with a chunk size of 50, ExRL achieved >90% success while all other baselines remained below 20%.
  • Rapid Real-World Adaptation:
    • Cube Stacking: Approached 100% success with only 10 minutes of physical interaction.
  • Clothing Folding: Reached ~90% success on a long-horizon task within 85 minutes.
  • Policy Agnostic: The method successfully adapted flow-based VLAs, autoregressive models, and world action models without needing task-specific hyperparameter tuning.

Limitations and Future Work

  • Action Space Constraints: If the base policy is completely incapable of proposing a necessary action (e.g., it never learned to open a specific type of gripper), execution length control alone cannot fix it. The authors suggest that their Online BC extension is a step toward solving this, but it wasn't the primary focus of the real-world tests.
  • Multi-Task Generalization: The current study focuses on adapting to one task at a time. Future work could explore whether an execution-length adapter can generalize across multiple different tasks or unseen objects.
  • Inference Latency: The paper does not deeply explore how adaptive replanning interacts with the computational latency of very large models (like VLAs), which is a critical factor for real-time control.

Summary updated

Abstract

Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
BibTeX
@misc{luo2026exrl,
  author = {Xu Luo and Jiaying Yang and Hao Wu and Junlin Xie and Zijin Hong and Ji Zhang and Lianli Gao and Jingkuan Song},
  title = {{ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control}},
  year = {2026},
  eprint = {2611.13482},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13482},
}
Figure