Robots trained by imitation copy what people showed them, and they cannot do better than those demonstrations. A recent fix keeps the imitation policy frozen, samples several candidate motions from it, and lets a separately learned value function pick among them. The value function is trained on the robot's own successes and failures, so the robot improves without new human data. The fix has a blind spot that matters most when the world moves. If every sampled motion arrives at the wrong moment, for example reaching where a handed-over object was half a second ago, then the value function has nothing good to choose. In moving scenes, the path the demonstrator took is often fine and the timing is what is wrong.
We propose to widen the candidate set with cheap, geometry-defined edits to the prior's proposals. These edits retime a candidate chunk (speed it up, slow it down, or insert a pause) and shift it rigidly along the predicted motion of the target or an obstacle. A value function then learns from real outcomes which edits work. Both the policy and the value function read 3D point clouds, meaning sets of surface points measured by a depth sensor. Simulation and real sensors produce nearly the same point clouds, so the edits and the learned values should carry over from simulation to hardware. In short, we let a learned critic choose among timing and offset corrections to an imitation policy's proposals, not only among the proposals themselves.
Core Claim
In dynamic manipulation, much of the gap between an imitation policy and task success is timing and phase error relative to moving objects, not a shortage of path shapes. A small, structured family of spacetime edits applied to frozen-prior proposals, scored by a point-cloud-history value function trained on autonomous rollouts, will exceed three things: the imitation ceiling, value-guided selection restricted to the prior's own samples, and a safety filter placed on top of the prior. The value-guided selection method is Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. The claim can be falsified, and three findings would falsify it. The first is that drawing more samples from the prior closes the gap. The second is that a collision-avoiding filter on the prior matches task success. The third is that the edits help in simulation but not after transfer.
Relation to Prior Work
The mechanism comes from Beyond Imitation. That paper freezes a large behavior-cloning policy, trains an off-policy value function (Q-function) on both successes and failures, and executes a Q-weighted average of sampled action chunks. It reports gains such as LIBERO-10 rising from 93% to 99% and real stack-cups rising from 40% to 90%. It works from 2D image tokens in static scenes. Its own stated limitation motivates our work: the critic can only promote actions the prior already proposes. Averaging over those proposals also keeps execution on the demonstration manifold, and that is exactly the wrong behavior when every proposal is mistimed.
The straightforward extension would be to run the same method on point-cloud features. We do not stop there, because that change alone does nothing about the exploration boundary. The new ingredient is the edit family and the requirement that the critic learn over edited candidates.
Two papers show that point clouds carry dynamic manipulation across the sim-to-real gap, but both remain bounded by their training signal. Learning Human-to-Robot Handovers from Point Clouds maps segmented point clouds to arm motion for handovers with a moving human and plateaus at 68% on real hardware, with no mechanism for improving from deployment failures. PointVLA injects point-cloud features into a frozen vision-language-action model and improves on a moving conveyor task, but it is trained purely by imitation.
TRANSIC goes beyond its point-cloud policy through a residual correction learned from human teleoperation. We replace the human corrector with autonomous outcome labels and target moving rather than static fixtures.
Safety-filter work constrains our design. Sailing Through Point Clouds builds control barrier functions directly from raw point clouds in under 1 ms. A control barrier function is a constraint that keeps the robot out of unsafe states. That paper also documents that such filters stall in front of obstacles unless they are paired with a parallel look-ahead. Sequential Neural Barriers learns barriers from the motion histories of many tracked agents. Both approaches avoid collisions, but neither makes a robot intercept a moving object on time, so a filter alone cannot solve our task.
We borrow one idea from Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting, which forecasts spacetime occupancy from raw sweeps without labels. We use its premise that motion can be predicted directly from geometry. We do not use its full model; our edits need only a velocity estimate over a short horizon.
Research Program
The prior is a point-cloud diffusion policy that outputs short action chunks, following Diffusion Policy. It is trained by imitation on HandoverSim-style demonstrations and then frozen. At each decision step we draw $N$ chunks $\tau_1,\dots,\tau_N$ from the prior and apply a fixed edit set $E$ to each. The edit set contains three kinds of edit: time-warps with rate $\alpha \in \{0.6, 0.8, 1, 1.25, 1.5\}$, a hold of $k$ control steps, and a translation $\delta = v\,\Delta t$, where $v$ is the target or obstacle velocity estimated by least-squares from centroids of the last few segmented point clouds and $\Delta t$ is a lookahead offset. This yields $N|E|$ candidates. The critic $Q_\phi(h_t, c)$ scores each candidate $c$ given the recent point-cloud history $h_t$, which is a short stack of frames and therefore encodes motion. We execute
$$c^\ast = \arg\max_{c \in \{e(\tau_i)\,:\, i \le N,\ e \in E\}} Q_\phi(h_t, c),$$
subject to a Vessel-style point-cloud barrier check that removes candidates predicted to collide. We use argmax over edits rather than a weighted average, because averaging mistimed chunks produces another mistimed chunk. This is the tension noted for Beyond Imitation.
The critic uses HL-Gauss categorical regression, as in that paper: value is predicted as a distribution over bins rather than as a single number. It is trained on all rollouts, successes and failures alike, from iterated deployment rounds. The reward is sparse task success, which HandoverSim provides. On real hardware we obtain it from the gripper's grasp state after the human releases the object. To keep inference latency bounded, we score candidates in one batched forward pass. We also measure how latency scales with $N|E|$ and report the knee of the success-versus-latency curve.
Baselines
The naive reference is the frozen point-cloud diffusion prior executed directly with receding-horizon replanning. The strongest published number on our main benchmark belongs to the point-cloud teacher-student policy of Learning Human-to-Robot Handovers from Point Clouds. It reports 68.75% simulated success on simultaneous-motion handovers and 68% real success, against 36.81% and 35% for GA-DDPG. We will re-run it from released or reimplemented code on the same splits and trial counts.
The strongest method-level competitor is Beyond Imitation ported to point clouds: the same prior, the same critic architecture, the same rollout budget, and the same number of scored candidates. It is scored with its Q-weighted average and, separately, with argmax, but without edits. This comparison isolates the edit set, the key ingredient.
The filter baseline is the prior with a point-cloud barrier filter in the style of Sailing Through Point Clouds. Filtered supervised fine-tuning retrains the prior on its successful rollouts only. The same paper reports this approach stalling at 55% and 30% on real tasks where Q-planning reached 90% and 80%.
Ablations use the edits without a learned critic, choosing edits by a hand-coded rule that matches predicted target arrival time, to test whether learning is needed at all. Further ablations remove the history input to the critic, so it sees a single frame, and restrict the edit set to retiming only or offsets only.
All learned methods share the same demonstrations, the same autonomous rollout count per round, and the same hyperparameter search budget. We count a win if our method beats the ported Q-planning baseline by more than the 95% confidence interval on the unseen-subject and unseen-object splits and beats 68% on real handovers.
Why not just…?
Why not just draw many more samples from the prior? More samples add diversity along directions the demonstrations already cover. Receding-horizon diffusion policies do recover from perturbations, but they cannot produce behavior outside their data, as argued in the file's discussion of Diffusion Policy. The real threat is that a large $N$ happens to include well-timed chunks. We test this directly by giving plain Q-planning $N|E|$ raw samples, matching our candidate count and latency. If it matches us, the edits are unnecessary and we will say so.
Why not just put a control barrier filter on the imitation policy? A filter prevents collisions but does not make the robot arrive on time. Filters also freeze in front of obstacles when used without look-ahead, as Sailing Through Point Clouds reports. We expect the filter baseline to reduce unauthorized contacts while leaving success near the prior's level or lowering it through timeouts. If it matches our success rate, the value function is not earning its place.
Why not just fine-tune the prior with reinforcement learning or on filtered successes? Filtered fine-tuning discards failures and stalled in Beyond Imitation. Reinforcement learning from scratch on handovers reached only 36.8% in HandoverSim. Our prior is small, though, so an honest residual-RL fine-tune, a TRANSIC-style residual trained on rewards instead of human corrections, remains a credible threat. We include it at matched rollout budget.
Proposed Experiments
The primary testbed is HandoverSim across all four splits: default, unseen subjects, unseen handedness, and unseen objects. We report success, drops, timeouts, unauthorized contacts, and execution time.
The second testbed is a moving-obstacle variant of a ManiSkill2 pick-and-place task that we construct. A second object sweeps through the workspace at randomized speeds. No reported result was found on Lacuna for this setting, so every number there will be ours.
A static control on LIBERO-10 checks that the edits do no harm where timing is irrelevant. The ported Q-planning baseline should land near the reported 99%, and ours should not fall below it.
Real transfer uses a Franka Panda with the 10 YCB objects from the point-cloud handover study, with 50 or more trials per method. We interpret real numbers with a known bias in mind: the simulated human is a non-reactive replayed giver, whereas real humans compensate for robot errors, which HandoverSim documents as 80% real versus 55% simulated. We therefore also run a scripted "uncooperative" real protocol, in which the giver holds a fixed trajectory, so that human compliance does not mask policy errors.
Three decisive checks follow. First, whether the gain over raw-sample Q-planning survives at matched candidate count; this is the central question. Second, whether the learned critic beats the hand-coded arrival-time edit rule; if it does not, the contribution reduces to a heuristic. Third, whether the ratio of real to simulated improvement is comparable to that of the point-cloud baseline, which would indicate the edits transfer as well as the geometry does.
References
- Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- Learning Human-to-Robot Handovers from Point Clouds
- PointVLA: Injecting the 3D World Into Vision-Language-Action Models
- TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
- Sailing Through Point Clouds: Safe Navigation Using Point Cloud Based Control Barrier Functions
- Sequential Neural Barriers for Scalable Dynamic Obstacle Avoidance
- Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers