Amortized Best-Response Refinement of Action Chunks for Real-Time Bimanual and Dexterous Control

Robots with many joints, such as two arms working together or a multi-fingered hand, increasingly run learned policies that output short "action chunks", meaning sequences of a few dozen future motor commands predicted at once. These policies are fast, but they execute open-loop. They have no way to correct a chunk that is about to drop an object or make two arms collide. Online optimization could repair such chunks, but searching jointly over every joint's chunk inside a 20 ms control cycle is the hard part, because the volume to search grows exponentially with the number of joints. We propose to split the robot into a few "blocks" (left arm, right arm, or palm and finger groups). On each control tick, each block re-optimizes only its own chunk while treating the other blocks' current chunks as fixed. The solution carries over and is improved on the next tick, rather than iterating to convergence inside one tick.

Multi-agent planning calls this pattern iterative best response: each player in turn picks its best move given the others' current moves. It is known to make otherwise intractable coordination problems solvable. Here we apply it inside a single robot, over temporally extended actions, under a hard real-time budget. The decomposition runs straight into one well-known failure: when two arms hold the same rigid object, their motions are physically locked together, and optimizing one while freezing the other produces internal stress or infeasible plans. The proposal's second ingredient addresses this. The block structure changes with the contact phase. When the arms share a grasped object, they merge into a single block whose decision variable is the object's motion, and both arms' commands follow from it.

Core Claim

At an equal per-tick compute budget, refining a learned chunk policy's proposals with one block-wise best-response sweep per tick, using contact-dependent blocks and warm-starting from the previous tick, will raise closed-loop success over two alternatives: executing the policy's chunks unrefined, and refining them with a centralized sampling optimizer over the concatenated chunk. The gap should widen as the number of controlled joints grows. The reasoning is simple. A sampling optimizer with $N$ samples in a $d$-dimensional chunk space covers a vanishing fraction of it once $d$ is large. Splitting into $m$ blocks gives each block $N/m$ samples in roughly $d/m$ dimensions, which is far denser coverage. Block coordinate descent pays for this with possible stalls at coordinate-wise stationary points. Contact-dependent merging and cross-tick warm-starting are meant to keep those stalls rare. All of this is an empirical bet, and the experiments are designed so it can fail cleanly.

Positioning Against Prior Work

The method draws on three lines of multi-agent work. Taming Decentralized POMDPs introduced JESP, which optimizes one agent while holding the others fixed and settles for a local equilibrium. At horizon 3 this took under 1 ms, versus 317,800 ms for exhaustive search. Continuous Time Planning for Multiagent Teams with Temporal Constraints folds frozen peers into an augmented single-agent problem and reports a roughly 412x speedup over a global solver. An Iterative Algorithm for Solving Constrained Decentralized Markov Decision Processes converges in fewer than four iterations on rover missions.

All three motivate the decomposition, but they also constrain it. Each relies on transition independence between agents and works with discrete, high-level tasks. Neither assumption holds for two arms on one object, which is why our block structure has to change with contact.

The closest work is Deep Interactive Motion Prediction and Planning. Its IBR-MPC alternates between an ego vehicle's plan and predicted plans for surrounding vehicles. It works over single-step controls of low-dimensional cars, and it notes that iterating to convergence is expensive enough to require cross-entropy sampling shortcuts. Our proposal differs in four ways: the players are parts of one high-dimensional body, the decision variables are multi-step chunks, the iterations are spread across control ticks rather than finished within one, and the player structure is not fixed.

Temporal abstraction supplies the action representation. Demystifying Action Space Design for Robotic Manipulation Policies shows that chunks expressed as offsets from the chunk's start state ("chunk-wise delta") keep error bounded, whereas step-by-step offsets accumulate error. Chunk-wise delta outperformed step-wise delta by about 10% on average. We adopt the chunk-wise delta parameterization because it makes shifting a chunk forward by one tick for warm-starting well behaved. That paper studies only feedforward imitation, with no online optimization.

Compositional Planning with Jumpy World Models plans over temporally extended actions with global random shooting. It reports roughly 80% success on AntMaze-Giant, where the base policies achieved 0%. That planner works offline at decision time, without a per-tick deadline and without any decomposition across actuators.

On the real-time side, Online Dynamic Trajectory Optimization and Control for a Quadruped Robot reaches 18.34 ms per iteration only by using a reduced model and a 2-step horizon. Efficient Trajectory Optimization for Robot Motion Planning needs 1–3 s for a single 6-axis arm. Both show that centralized optimization of full chunks in high dimension does not fit a 20 ms budget without strong simplification.

Finally, the high-DoF learning systems serve as the proposal generators we refine and as the baselines we must beat. DECO generates bimanual dexterous action chunks with a diffusion transformer. ADEPT trains monolithic reinforcement learning policies for 23–29 DoF arm-hand systems.

Research Program

The system has three parts.

The first is a chunk proposal policy. It is a diffusion policy trained by imitation on RoboTwin 2.0 demonstrations, outputting chunk-wise delta chunks of length $H$ for each block. Its samples define the initial plans and the sampling distribution around them.

The second is a block-wise refinement step. Let $c_i$ denote block $i$'s chunk, $c_{-i}$ the other blocks' current chunks, and $J$ a rollout cost combining task progress, collision, joint limits, and object-drop penalties, evaluated in parallel simulator rollouts. Each tick performs one sweep:

$$c_i^{t} = \arg\min_{c_i \in S_i^t} J(c_i, c_{-i}^{\text{latest}}), \quad i = 1,\dots,m,$$

where $S_i^t$ is a set of $N/m$ samples drawn around block $i$'s shifted previous solution and around fresh policy proposals. "Latest" means blocks updated earlier in the same sweep are already used. Only the first few commands are executed, and the rest of each chunk is shifted forward to warm-start the next tick. This is the multi-player analogue of performing one optimizer iteration per control cycle.

The third is contact-dependent partitioning. A contact detector reads the simulator, or on hardware the tactile signals that DECO exposes. When both end-effectors grasp the same object, the arm blocks merge into a single block that samples the object's pose trajectory, and each arm's commands follow by inverse kinematics. Fingers that pinch jointly merge in the same way.

For the dexterous setting, we will build a 23-DoF arm-hand simulation modelled on ADEPT's Kuka-Allegro system. Its blocks are the arm, the thumb, and the remaining fingers, which tests whether the advantage scales with $d$.

We will use the true simulator as the rollout model first. This is privileged information, so the result is an upper bound on what the method can achieve. A second phase replaces it with a learned chunk-level dynamics model, where errors in that model become an additional confound that we will measure separately.

Baselines

The naive reference is the chunk proposal policy executed open-loop, with no refinement. This is how ACT, Diffusion Policy, and DECO-style systems are actually deployed, and it isolates the value of online refinement at all. It uses the same trained network as our method, so the comparison is exact.

The strongest competitor is centralized sampling refinement of the concatenated chunk. This is MPPI, an optimizer that perturbs a nominal plan with random noise and averages the samples weighted by their cost, initialized from the same policy proposals and warm-started across ticks in the same way. It is the strongest alternative because it optimizes the true coupled objective with no decomposition error. If any refinement method should win, it is this one.

Fairness is enforced by giving it the same rollout model, the same total number of rollouts per tick, the same horizon, and the same wall-clock budget on the same GPU. Each method gets an equal hyperparameter search (noise scale, temperature, sample split) on held-out seeds.

For reference numbers: on RoboTwin, the file reports no closed-loop success rate for diffusion-policy chunks per task; no reported result found on Lacuna for MPPI-refined chunk policies on RoboTwin or DECO-50. The hardware context we inherit is DECO's 72.25% vision-only and 82.50% tactile-adapted average success on DECO-50, against ACT's 57.25%. Beating the centralized baseline means a statistically significant gain in closed-loop success, measured over at least 10 RoboTwin bimanual tasks with many seeds, at matched latency. On the 23-DoF hand, the gain should be at least as large as on bimanual tasks.

Three ablations isolate the ingredients. Fixed partitioning keeps the arms as separate blocks even when they share an object, testing whether contact-dependent merging is what handles physical coupling. Converge-within-tick runs several sweeps per tick with proportionally fewer samples per sweep, testing whether amortizing across ticks matters. A fixed leader-follower scheme lets one block plan while the other only responds and never updates the leader, testing whether mutual iteration matters.

Why not just…?

Why not just train a bigger end-to-end chunk policy and skip online optimization? Monolithic learned policies remain far from saturation on contact-rich bimanual tasks. On DECO-50, ACT reached 57.25% and Diffusion Policy 51.25%, and even DECO with tactile input reached 82.50%. The CompPlan results show that adding decision-time search on top of fixed policies can turn 0% into roughly 80% on long horizons. Our method keeps the learned policy and adds refinement, so the open-loop baseline directly measures whether refinement helps. If the gain over unrefined chunks is negligible on RoboTwin, the case for the method collapses, and we will report that.

Why not just run centralized MPPI over all joints, given that GPUs make sampling cheap? This is the real threat. With a good learned proposal distribution, the effective search dimension may already be small enough that centralized sampling suffices. The matched-budget comparison settles it. We predict that on 14-DoF bimanual tasks the gap may be modest, and that it opens on the 23-DoF hand. If centralized MPPI matches block-wise refinement on both, the decomposition buys nothing, and we would redirect the work toward the contact-merging idea applied to centralized sampling.

Why not just iterate best response to convergence each tick, as IBR-MPC does? IBR-MPC itself reports that this is expensive in low-dimensional driving. In high dimension within 20 ms, iterating to convergence leaves few samples per sweep. The converge-within-tick ablation tests directly whether one sweep per tick with warm-starting is the better use of the budget.

Why not just use a consensus method such as ADMM to handle the arm coupling? Consensus splitting does handle coupling in principle, but it introduces dual variables that need many iterations to settle. That conflicts with a per-tick budget, and it is designed for gradient solvers that struggle with contact discontinuities. We treat it as a secondary comparison on the shared-object subset of tasks, where it is most relevant.

Proposed Experiments and Decisive Checks

The main experiment runs RoboTwin 2.0 bimanual tasks in closed loop, choosing tasks that require both independent and shared-object phases. We will report success rate, collision and drop rate, and the 99th-percentile solve time against a 20 ms deadline.

The scaling experiment repeats the comparison on the simulated 23-DoF arm-hand, using peg-insertion and reposing tasks modelled on FMB and VisDex. It also includes a controlled sweep in which we artificially vary the number of actuated fingers, and plots the gap between block-wise and centralized refinement against $d$.

The coupling diagnostic logs the internal force between the two grippers during shared-object phases under fixed and contact-dependent partitioning.

The decisive checks are three. First, the method must beat open-loop chunks; otherwise refinement is pointless. Second, it must beat matched-budget centralized MPPI by a margin that grows with $d$; otherwise the decomposition adds nothing. Third, contact-dependent merging must remove the coupling failures that fixed partitioning shows; otherwise Tension 2 defeats the method as designed.

A hardware stage on a dual-arm platform, using DECO-50 demonstrations for the proposal policy, follows only if all three checks pass in simulation.