Reinforcement learning from a fixed dataset of robot trajectories works best when the policy commits to a short sequence of future actions, called an action chunk, instead of choosing one action at a time. Committing to a chunk lets the value estimate (the learned prediction of eventual success) pass a single end-of-task reward back over many steps at once, and it keeps exploration from dissolving into jitter. The cost is that the robot executes each chunk blind. If a block slips at step two, the remaining steps replay as if nothing happened. We propose a different kind of chunk. Instead of a list of actions, each chunk is a small controller: a planned sequence of states, the actions that produced them in the data, and a learned set of correction gains that push the robot back toward the plan whenever it drifts. The chunk is still a commitment, so the value learning is preserved, but it reacts to errors inside the commitment.
This borrows the oldest idea in control engineering. A feedforward path supplies the bulk command, and a feedback path cleans up the error that remains. We apply it to the specific place where chunked offline reinforcement learning is known to break. The claim that makes the idea testable is this: on dataset trajectories, a closed-loop chunk outputs exactly the recorded actions, so it can be trained with the same value targets as an ordinary chunk. When deployment deviates from the data, the feedback pulls the robot back toward states the value function has actually seen. That should shrink the gap between what the value function predicts and what the robot achieves, a gap that current methods identify but do not correct.
Core Claim
Reinforcement Learning with Action Chunking (Q-chunking) trains a value function over whole chunks of length $h$ and reports 74% on OGBench cube-quadruple, where one-step methods score 0%. Its chunks run open-loop. Decoupled Q-Chunking (DQC) keeps a long-horizon value function but distills a partial one, so the policy only needs to emit a short prefix of $h_a \ll h$ steps. DQC reaches 98% on cube-triple-100M. Its authors also name an "open-loop value bias": values learned from recorded action sequences assume the recorded state rollout will recur, which fails under stochastic dynamics. They note that DQC inherits this bias. Both papers treat reactivity as a matter of chunk length.
We treat it as a matter of chunk structure. Let $s_t$ be the observed state and let a chunk be the triple $c = (\hat s_{1:h}, \bar a_{1:h}, K_{1:h})$. Here $\hat s_k$ are reference states, $\bar a_k$ are nominal actions, and $K_k$ are feedback gain matrices. Step $k$ of the chunk executes
$$u_k = \bar a_k + K_k\,(s_k - \hat s_k).$$
Suppose a chunk is read off a dataset segment, with $\hat s$ and $\bar a$ set to the recorded states and actions. Along that segment $s_k = \hat s_k$, so $u_k = \bar a_k$. The chunked value target used by Q-chunking is therefore unchanged by adding feedback, and no new off-policy correction is needed. The gains matter only off the data, and there they act to return the state toward $\hat s_{h}$, where the value function's bootstrap target was learned.
The prediction follows from this reasoning rather than from a theorem. The open-loop value bias should shrink as feedback reduces the spread of end-of-chunk states around the reference. With $K = 0$ the method reduces to Q-chunking. Conversely, if no deviations occur at test time, feedback should neither help nor hurt.
Research Program
The central problem is where the gains come from without a dynamics model and without more demonstrations. We identify them from the dataset itself.
OGBench explicitly shows that noisy, exploratory data is what makes offline learning work: pick-and-place success falls from 99% to 6% without noise, according to OGBench. That noise means the dataset contains many pairs of nearby segments that start close together, diverge slightly, and are followed by different actions. We train a gain network $G_\phi$ to predict time-varying, low-rank gains $K_{1:h}$ from the start state and the nominal chunk. It is trained to explain action differences between such paired segments:
$$\min_\phi \; \mathbb{E}_{(c,c')}\;\textstyle\sum_{k}\big\| (\bar a'_k - \bar a_k) - K_k(\hat s'_k - \hat s_k)\big\|^2 .$$
Here $(c, c')$ are dataset segment pairs whose start states lie within a small radius. The objective recovers the corrective reflex already implicit in the data-collecting behavior, which is a data-driven analogue of fitting a time-varying linear feedback law.
A second variant then adjusts only the gains, holding $\bar a$ and $\hat s$ fixed, to maximize the chunked value under state perturbations. The perturbations are sampled deviations applied inside a learned one-chunk state predictor. This variant tests whether gains tuned for value beat gains that merely imitate the behavior's reflexes.
The rest of the system reuses DQC's machinery. The policy is a flow-matching model that proposes the reference-and-action part of the chunk, several candidates are ranked by the chunked value function, and the long critic plus the distilled partial critic follow DQC. The one structural change is that execution runs the feedback law at every control step, while the network is queried only once per chunk. Feedforward and feedback therefore run at different rates. This echoes FAVLA, which pairs a slow model with a fast force-conditioned action head. FAVLA is pure imitation and, by its own evaluation, sometimes misreads misalignment forces as successful insertion. Our fast path is instead a correction law that answers to a value function.
The gains also give a natural signal for adaptive commitment. Chunk horizons are fixed in DQC, and Mixture of Horizons picks prefix length from disagreement between horizons without any value awareness. We can end a chunk early when the tracking error $\|s_k - \hat s_k\|$ exceeds what the gains were fit to correct. This is a secondary study. The main result must stand with a fixed horizon.
The analysis has two parts. First, we measure open-loop value bias directly as the gap between the predicted chunk value and the realized return from many rollouts at matched start states, plotted against injected perturbation strength. Second, we measure end-of-chunk state spread with and without feedback. If the mechanism is real, the reduction in value bias should track the reduction in spread.
Baselines
The naive reference point is Q-chunking with open-loop execution of full chunks. It shows the cost of blindness under the same value learning. Its reported figures are 17% on cube-triple-100M and 29% on cube-quadruple-100M, as cited in Decoupled Q-Chunking. We also include DQC-naive, which executes the first steps of a full-chunk policy and falls from 98% to 36% on cube-triple. It shows that simply cutting a chunk short is not reactivity.
The strongest existing method is DQC itself. It reports 98% on cube-triple-100M and 93% on cube-quadruple-100M in Decoupled Q-Chunking, against 82% and 67% for SHARSA. It is the right competitor because it already attacks exactly our problem, reactivity without losing the benefits of chunked value learning, by shortening the executed horizon. We run it from the authors' configuration with $h_a \in \{1, 2, 4\}$. Our method receives the same critic horizon, the same datasets, the same gradient-step budget and the same number of tuning sweeps. $h_a = 1$ is DQC's most reactive setting and must be included.
On the unperturbed 100M tasks, we do not expect to beat 98%, and that is not the claim. Beating DQC means matching it within noise on clean evaluation while losing clearly less success as perturbation strength grows. It also means doing so at a lower network query rate than DQC at $h_a = 1$.
Two ablations isolate what is new. The first replaces the learned gains with a fixed, hand-tuned diagonal proportional gain around the reference states, which is the standard trick of tracking predicted waypoints with an impedance controller. It tests whether learning the gains matters. The second keeps the learned gains but trains the value function without them, then turns feedback on only at deployment. It tests whether it matters that the critic evaluates the closed-loop chunk, which is our central claim, rather than feedback merely acting as a safety net.
For dynamic, response-sensitive manipulation we add an every-step reactive imitation policy, Responsive Noise-Relaying Diffusion Policy (RNR-DP). It reports an 18% success improvement over Diffusion Policy on ManiSkill Relocate and RollBall, and it shows what full reactivity without value learning buys.
Why not just…?
Why not just run DQC with a one-step actor? This is the real threat. At $h_a = 1$ DQC re-queries its policy every step and is fully reactive. Its partial value function, however, is trained to assume optimal completions of recorded action sequences, and it still carries the open-loop value bias its authors describe. Each correction is also a full network query. If DQC at $h_a = 1$ matches our perturbed-evaluation success and shows equally small measured value bias, then the value-side argument is wrong. In that case our contribution shrinks to a reduction in query rate, and we would report it as such. The perturbation sweep and the value-bias plot are designed to settle exactly this.
Why not just track the predicted waypoints with a fixed PD controller? This is common robotics practice, and it may capture most of the gain on simple reaching. A single gain is wrong across phases, though. Minimum Attention Optimal Control Laws show that optimal schedules lean on feedforward during gross motion and raise feedback gains only near the terminal target. FAVLA shows that rigid execution raises peak contact force from 7.7N to 12.0N in gear assembly. The fixed-gain ablation tests this directly. If it ties learned gains, we drop the gain network and keep only the claim about training the critic with closed-loop chunks.
Why not just use shorter chunks or every-step imitation such as RNR-DP? Shorter chunks trade away the commitment that makes sparse-reward learning work: one-step methods score 0% on cube-quadruple in Reinforcement Learning with Action Chunking. Imitation-only reactivity cannot stitch suboptimal data, as the OGBench 99%-to-6% collapse shows. Imitation can succeed on clean human data, as with BC-RNN at 84% on Square in What Matters in Learning from Offline Human Demonstrations, so we include RNR-DP and report where it wins.
Proposed Experiments
The primary testbed is OGBench cube-double, cube-triple, cube-quadruple, scene and puzzle, all with sparse goal rewards. We evaluate at both the standard 1M-transition size and the 100M size, to test behavior under data scarcity. No reported result was found on Lacuna for DQC at the 1M dataset size, so we run it ourselves.
Because the benchmark's default dynamics may leave little room for open-loop failure, we add evaluation-time perturbations of graded strength. These are Gaussian action noise, random impulses to held cubes, and friction randomization. We report the success-versus-strength curve alongside clean success. No reported result was found on Lacuna for any chunked RL method under such perturbations, so the whole curve is new measurement.
The secondary testbed is ManiSkill Relocate and RollBall from demonstrations with sparse success reward. Here RNR-DP's 18% margin over Diffusion Policy is the imitation reference.
The metrics are success rate, the measured open-loop value bias (predicted minus realized return), end-of-chunk state spread, and network queries per second.
The decisive checks are three. At zero perturbation, closed-loop chunks must match DQC, because on-data equivalence predicts no clean-condition penalty. Under perturbation, the success margin must grow with strength, and value bias must fall in step with state spread. The critic-without-feedback ablation must do worse than full training, which shows that evaluating closed-loop chunks, not only executing them, is what matters.
A result showing large success gains with unchanged value bias would mean feedback is only a safety net, and we would say so. A result where DQC at $h_a = 1$ matches us on every metric except query rate would narrow the claim to efficiency.
References
- Reinforcement Learning with Action Chunking
- Decoupled Q-Chunking
- OGBench: Benchmarking Offline Goal-Conditioned RL
- FAVLA: A Force-Adaptive Fast-Slow VLA model for Contact-Rich Robotic Manipulation
- Mixture of Horizons in Action Chunking
- Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Control
- On the Existence and Computation of Minimum Attention Optimal Control Laws
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation