Measuring Credit in Task Progress, Not Clock Ticks, When Demonstrations Are Imperfect

Robots trained from human demonstrations inherit whatever the humans did. Much of what people do while teleoperating a robot is harmless slowness, such as pausing, hesitating or moving at an uneven pace. Some of it is real error, such as a misaligned grasp that causes a drop forty steps later. Current methods for learning from mixed-quality demonstrations measure everything in clock steps, so they cannot tell these two apart. A slow but correct demonstration scores worse than a fast but lucky one, and blame for a late failure is spread over hundreds of idle frames. We propose to re-index every demonstration by a learned task-progress coordinate: a number between 0 and 1 that rises only when the task actually advances and holds still when the demonstrator pauses. We then do all credit assignment in steps of progress rather than steps of time. The same coordinate also runs online at deployment, so a robot that is blocked mid-task simply stops advancing instead of drifting out of distribution.

The core claim is that one learned progress variable, trained from demonstrations that carry only a success-or-failure label at the end, does three jobs that the literature currently handles separately. It makes the quality estimates of demonstrations insensitive to tempo. It shortens the chain over which a terminal outcome must be propagated backward, from hundreds of clock steps to a few dozen progress increments. It gives the policy a state-dependent clock of the kind that Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation showed is essential for recovering from disturbances. That work needs an optimal planner trajectory to define it; we do not.

Where this sits

Four lines of prior work surround this idea, and each leaves out one piece. The loco-manipulation work of Sleiman et al. supplies the mechanism. Its clock variable advances with tracking quality along a single trajectory-optimized reference, and it reports 96.33% success on a spring-loaded door pull versus 15.38% for a linear-clock baseline. Its reference is assumed optimal, however, and its reinforcement learning uses ordinary step-wise updates. Classical phase formulations, such as the phase-velocity movement representation of Auke Ijspeert et al. and the recursive phase filter of Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives, likewise assume demonstrations define the target behavior. None of them asks which demonstrations to trust.

On the imperfect-demonstration side, Exponentially Weighted Imitation Learning for Batched Historical Data (MARWIL) reweights cloning by exponentiated advantage. Its advantage is built from clock-indexed discounted returns, which is exactly where tempo contaminates quality. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation motivates our design and also constrains it. It found that value-based offline methods (BCQ, CQL) collapse to roughly 0–10% on human Square and Transport data while a recurrent cloning policy (BC-RNN) reaches 84% and 71.3%. We take this as a warning that any value function we train must be kept small and low-horizon, and that it must be judged against BC-RNN, not against offline RL.

On credit assignment, Hindsight Credit Assignment asks which past actions made an outcome likely, but was shown only on discrete diagnostic tasks. Beyond Simple Sum of Delayed Rewards (CoDeTr) redistributes delayed rewards with attention over a clock-time token grid, so pauses consume its capacity. Scaling data-driven robotics with reward sketching and batch reinforcement learning comes closest in spirit, because its humans draw progress curves over videos. That progress is a retrospective reward drawn by annotators, not a learned coordinate that the policy runs on. We borrow its insight that progress is the right annotation and remove the annotator.

Research Program

The first component learns the progress coordinate $\phi(s) \in [0,1]$ from successful demonstrations, with no planner and no hand-drawn curves. Within each successful demonstration, we train a network with a ranking loss that asks later states to score higher than earlier ones. The margin of this loss is scaled by how much the relevant state actually changed between the two frames: object pose plus end-effector pose. Idle frames therefore receive no pressure to increase. Across demonstrations, a cycle-consistency term asks states matched as nearest neighbours in a learned embedding to receive similar $\phi$. This lets a slow demonstrator and a fast one share a progress scale. We deliberately restrict evaluation to tasks with a shared stage structure (Lift, Can, Square, Transport). The file's own open problems note that a scalar phase cannot represent branching strategies, and we treat that as a stated limitation rather than claim to solve it.

The second component does credit assignment in progress steps. For each frame $t$, let $\tau_\Delta(t)$ be the first later frame at which $\phi$ has risen by at least a fixed increment $\Delta$ (for example $0.05$). If the demonstration ends first, $\tau_\Delta(t)$ is the terminal frame. We train a success estimator $V(s)$ by bootstrapping across these progress jumps, without discounting, with the terminal target equal to the binary success label. The progress-indexed advantage of the segment starting at $t$ is then

$$A_\phi(t) = V(s_{\tau_\Delta(t)}) - V(s_t),$$

the change in estimated success probability over one unit of task progress. Three consequences follow. A pause collapses into a single segment, so it is neither rewarded nor penalized for duration. A fumble followed by a recovery is absorbed into one segment, and that segment's advantage reflects whether the recovery worked. The backward chain has roughly $1/\Delta$ links instead of hundreds, which should make this small value function far less brittle than the step-wise critics that failed in Mandlekar et al.

The third component is the policy. It is a recurrent cloning policy conditioned on the online estimate of $\phi$, trained with MARWIL-style weights $\exp(\beta A_\phi)$. Training frames are sampled uniformly in progress rather than in time, so long pauses are not overrepresented. Pauses are also not deleted: if waiting at some progress level precedes success in good demonstrations and is skipped in failures, the advantage will say so. At deployment, $\phi$ is recomputed from observations at every step. If the robot is blocked, $\phi$ stops rising and the policy stays at the matching point in the task, which is the behavior Sleiman et al. obtain with a planner reference. As a secondary test of the reward-engineering claim, we will use the learned $\phi$ in their task-agnostic reward form (reward for progress rate minus deviation) to fine-tune the offline policy online in simulation, replacing any hand-written stage rewards.

Baselines

The naive reference point is BC-RNN trained on all demonstrations with uniform weights, following What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. It is the strongest competitor on human teleoperation data, reporting 84% success on Square and 71.3% on Transport in simulation. We will rerun it with the published architecture on identical dataset splits and give it the same hyperparameter search budget as our method. Beating it means a statistically reliable improvement in success rate on the mixed-quality splits, plus shorter completion time, without losing success on the proficient splits.

The strongest reweighting competitor is MARWIL applied to the same BC-RNN backbone. We run it in two forms: with clock-indexed discounted returns built from the terminal success label, and with an undiscounted success return. The second form is the most dangerous rival, discussed below. Both share our backbone, $\beta$ grid and value-network size, so the only difference is how the advantage is indexed.

For stitching, we run IRIS from IRIS: Implicit Reinforcement without Interaction at Scale on Lift and Can. It reports 81.3% on Lift, where plain BC reports 13.7%, and 28.3% on RoboTurk Cans, where baselines report 0%. We give it the same success labels as its reward.

For clock-indexed credit redistribution, we run CoDeTr on the same data to produce dense per-step rewards, then use them for advantage weighting. There is no reported result on Lacuna for CoDeTr on robomimic or RoboTurk tasks, so we establish its numbers ourselves.

Four ablations isolate the key ingredient. The first swaps the learned $\phi$ for normalized time $t/T$. The second swaps it for progress obtained by dynamic-time-warping each demonstration to a medoid successful demonstration. The third drops the displacement-scaled margin, so idle frames are forced to increase. The fourth keeps $\phi$ for policy conditioning but computes the advantage in clock steps. The fourth ablation is the cleanest test of whether indexing credit by progress, rather than merely having a phase input, is what helps.

Why not just…?

Why not just run MARWIL with an undiscounted success return? Removing the discount does remove the penalty on slow demonstrators, and this is a genuine threat to the tempo argument taken alone. What it does not do is localize blame. A Monte Carlo success label assigns the same advantage to every frame of a failed demonstration, including the good opening. A bootstrapped clock-step critic must chain hundreds of steps, the regime where Mandlekar et al. saw value methods collapse. Our localization experiment below directly compares the two on data where the causal error is at a known progress point. If undiscounted MARWIL matches us there too, the credit-assignment claim is wrong, and we will report that.

Why not just trim pauses with a velocity threshold and run BC-RNN? On synthetically injected pauses this may work nearly as well, and we expect it to be competitive there. It fails in two situations. The first is necessary waits, such as letting a peg settle or a contact stabilize; a threshold deletes these indiscriminately, while progress-indexed advantage keeps them when they precede success. The second is deployment-time blockage, where no preprocessing of training data gives the policy a clock that stops. Both cases are measured separately below, so a win on injected pauses alone would not count for us.

Why not just use normalized time or dynamic time warping as the progress label? Normalized time assigns progress to idle frames, and warping to a single reference distorts demonstrations that take different valid routes. Warping also runs offline over complete trajectories, so it cannot run in closed loop, which is the gap highlighted by the 15.38% rigid-clock result in Sleiman et al. These are exactly ablations one and two. If the learned $\phi$ merely recovers $t/T$, those ablations will match our method and the progress-learning component is unnecessary.

Why not just rely on BC-RNN's hidden state to absorb tempo? The recurrent state does capture non-Markovian human behavior, but it has no mechanism for distrusting failed demonstrations, and nothing forces it to hold still when the robot is physically stopped. The blockage test and the failure-contaminated splits address this directly.

Proposed Experiments

We use the robomimic/RoboTurk Proficient-Human and Multi-Human datasets for Lift, Can, Square and Transport. We add two controlled corruptions with known ground truth. In the tempo corruption, random pauses and slowdowns of 1–5 seconds are inserted into successful demonstrations. In the causal-error corruption, a fraction of demonstrations are replaced by rollouts with a perturbed grasp at a known stage, and these later fail. Metrics are task success rate, completion time, and three diagnostic quantities.

The first diagnostic check is tempo invariance. The rank correlation between a demonstration's weights before and after pause injection should stay near one for our method and drop for clock-indexed MARWIL and CoDeTr. The second is localization. The fraction of negative advantage mass falling within the segment containing the injected grasp error should exceed the uniform-spread baseline by a wide margin, and should beat undiscounted MARWIL. This is the check that could falsify the central claim. The third is deployment blockage. In simulation we freeze the object or arm for 2–10 seconds mid-task and measure success after release, comparing our online $\phi$ against BC-RNN and IRIS.

The end-to-end test is success on Multi-Human Square and Transport against the reported 84% and 71.3%, and on Can against IRIS's 28.3%. The secondary test is online fine-tuning in simulation with the learned-progress reward versus the sparse success reward, measured as environment steps needed to reach a fixed success rate. The idea fails if the learned $\phi$ is indistinguishable from normalized time, or if undiscounted clock-indexed advantage localizes errors equally well. Either outcome would show that the progress coordinate adds nothing beyond the simpler alternatives.