Using the Learned Plan Graph to Grade Messy Demonstrations

Robots can learn long, multi-step tasks quickly when they first turn their demonstrations into a small symbolic vocabulary. That vocabulary consists of facts such as "block A is on block B" and actions that change those facts. Short neural controllers then carry out each action. Methods that discover this vocabulary automatically from robot demonstrations work with around a hundred examples. They also assume every demonstration is clean. Real teleoperated data contains failed grasps, regrasps, hesitations and detours. Fed such data, the discovery step treats each fumble as a meaningful event and invents spurious facts and actions. The resulting planner is broken before any controller is trained.

We propose to let the symbolic layer clean its own input. Once demonstrations are mapped into abstract states, a fumble becomes visible as structure: a loop (grasp, drop, grasp again), a dead end, or a path longer than one another demonstrator took between the same two abstract states. The shortest-path distance in the graph built from all demonstrations therefore gives a reward-free measure of how good each abstract step was. We use it to down-weight bad segments, re-learn the symbols on the reweighted data, and repeat. The aim is a hybrid planner-plus-controller that keeps the data efficiency of symbolic imitation without assuming the demonstrator was an expert, and without rewards or human rankings.

Core Claim

Each direction contributes one piece. Expert optimality is the assumption we remove, hybrid symbolic-neural control is the mechanism, and sample efficiency is what we measure. Our claim is that the induced abstract graph supplies an advantage signal for free. Define a cost of one per abstract step. Let $d(z, g)$ be the shortest-path length in the graph from abstract state $z$ to the goal abstract state $g$. For a demonstrated abstract transition $z \to z'$, the abstract advantage is

$$A(z, z') = d(z, g) - d(z', g) - 1,$$

which equals zero on a shortest path and is negative for detours and loops. Each demonstration segment then receives the weight $w = \exp(\beta A)$, where $\beta \ge 0$ is a temperature.

This is exactly the exponential reweighting of Exponentially Weighted Imitation Learning for Batched Historical Data (MARWIL). MARWIL multiplies the imitation loss by $\exp(\beta \hat A)$ and gets $\hat A$ from logged rewards and a learned value network. Here the value is the negated graph distance $-d(z,g)$, which costs nothing to compute once the graph exists. Because the graph merges abstract states across demonstrations, the distance also stitches demonstrations together: a segment from one person can be judged good because another person's segment completes the shortcut. The new ingredient is applying this weighting inside symbol discovery and alternating the two. No paper in our search does this.

Research Program

The base system is Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learning (NSIL), which supplies the method we modify. NSIL works in four steps:

  1. It clusters relative object features into predicates.
  2. It identifies the operators that cause predicate changes.
  3. It selects a symbol set with a complexity-regularized objective.
  4. It segments demonstrations by operator to train behavioral-cloning skills and subgoal samplers.

NSIL assumes clean demonstrations. Transferring Hierarchical Structures with Dual Meta Imitation Learning motivates the fix: it reports that unfiltered suboptimal demonstrations produce non-semantic primitives in purely neural skill hierarchies. We expect the same failure in symbolic discovery and will measure it.

The algorithm alternates two steps, initialized with all weights set to one.

  1. Abstract and grade. Run weighted NSIL induction to get predicates, operators and the abstract graph. Take the goal abstract state as the final abstract state of the successful demonstrations. Compute $A$ for every segment.
  2. Reweight and refit. Assign weights $w$ and refit predicate clustering, operator precondition and effect sets, and the complexity objective, with each transition counted in proportion to its weight.

Spurious predicates that exist only to explain fumbles lose support as their segments are down-weighted. The complexity penalty then removes them, which collapses the loops that created them.

Segments with negative advantage are not discarded. A "fumble, then recover" segment whose recovery ends in a valid operator effect is relabeled as recovery data for that operator's precondition, in the spirit of the recovery paths extracted by Robust Offline Imitation Learning Through State-level Trajectory Stitching. This targets compounding execution error, a known weakness of hierarchical controllers.

The low-level skills receive a second, local grade. Within one operator, each segment's time to reach the effect is compared to the median for that operator, and the behavioral-cloning loss is weighted accordingly. Because these horizons are short, this estimate is far less noisy than a global value network.

The analysis asks four questions:

  • Does the alternation converge to the symbol set NSIL finds on clean data? We score predicate and operator recovery against the clean-data domain.
  • How does the advantage behave when the dynamics are stochastic and a retry was actually necessary? In that case the retry segment should end up as recovery data rather than be suppressed.
  • How many demonstrations are needed at each fumble rate?
  • When does the graph itself become too corrupted to grade anything? This is the regime where the first-round abstraction has more spurious states than true ones.

Baselines

The naive reference point is NSIL run unchanged on the noisy data. This measures how badly fumbles damage discovery and sets the floor every method must clear.

The strongest flat competitor is IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data. IRIS proposes reachable subgoals with a conditional variational autoencoder, scores them with a batch-constrained Q-network, and executes them with a goal-conditioned behavioral-cloning controller. It is the best reported offline method on fumbled manipulation data:

  • Robosuite Lift (137 fumbled human demonstrations): 81.3% success, against 13.7% for behavioral cloning and 18.0% for batch RL.
  • RoboTurk Cans: 28.3% from low-dimensional state and 42.3% from images, against 0% for behavioral cloning and batch RL.

We will run IRIS from its published settings on identical demonstration sets, with the same tuning budget (a fixed grid for the goal horizon $T$, its most sensitive hyperparameter) and the same evaluation episodes. IRIS needs task rewards and our method does not, so we will report both reward-available and reward-free conditions honestly. Beating IRIS means matching its 81.3% on Lift and exceeding its 28.3% on Cans from state. On the multi-object Building tasks, where flat and latent hierarchies are expected to degrade, we should win by a clear margin.

For a conservative offline RL reference we run Conservative Q-Learning for Offline Reinforcement Learning (CQL) with the same reward access. CQL's paper reports that it alone consistently beat behavioral cloning on AntMaze, Adroit and Kitchen, but the file gives no per-task numbers, so we treat it as a strong but unquantified reference.

For a reward-based weighting of the same symbolic pipeline we run NSIL with MARWIL weights, using sparse task reward and a learned value network. For a ranking-based weighting we run Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality (CAIL) confidences, computed with its 5% ranked subset and plugged into NSIL. CAIL's flat policy reached an expected return of nearly 4000 on mixed-quality MuJoCo Ant. Both of these are given strictly more supervision than our method.

Three ablations isolate the key ingredient:

  • graph-distance weighting applied once, without re-induction;
  • loop removal alone, with no distance-based advantage;
  • the full alternation with the recovery relabeling turned off.

Why not just…?

Why not just throw away failed or unusually long trajectories, then run NSIL? This is the most serious threat. If fumbles are concentrated in a few demonstrations, trajectory-level filtering could match our method at almost no cost. Its weakness is that crowdsourced data tends to spread small errors across nearly every trajectory, as in the intentionally fumbled Lift set used by IRIS. Filtering then either keeps the fumbles or discards most of the data. We will vary the fraction of trajectories containing at least one fumble from 10% to 100% at a fixed per-trajectory fumble count. If length filtering keeps pace at high contamination, the segment-level machinery is not needed and we will say so.

Why not just run CQL or MARWIL on a flat policy? Both need a reward signal and a calibrated value function. MARWIL's own analysis ties its improvement to value accuracy. In long-horizon tasks, flat and hierarchical neural policies collapse where symbolic structure does not. SLAP reports 0% for pure and hierarchical RL against 100% for its symbolic method on Obstacle Tower (SLAP: Shortcut Learning for Abstract Planning). NSIL reports that hierarchical neural baselines fall toward zero as block count grows while its planner stays near perfect. The Building scaling experiment tests whether this gap survives noisy data.

Why not just weight NSIL's segments with MARWIL advantages from task reward? This is the obvious combination, and we run it as a baseline. It needs rewards and a global value network, and its weights are computed once on a fixed and possibly corrupted abstraction. Two outcomes would favor our design over it: matching its success without rewards, and showing that re-induction recovers predicates it cannot.

Why not just prune exact loops in the raw state trajectory? Continuous states almost never repeat exactly, so raw-space loop detection mostly finds nothing. The loop-removal-only ablation, run in abstract space, measures how much of the gain comes from cycles alone and how much from distance-based grading of detours.

Proposed Experiments

The main testbed is the set of Robosuite tasks used by NSIL (Painting, Pouring, Building), plus the fumbled Robosuite Lift and RoboTurk Cans data used by IRIS. For Robosuite we build controlled corruptions of clean scripted and teleoperated demonstrations by inserting failed grasps, regrasp loops, detours toward irrelevant objects, and pauses, at graded rates.

A second testbed is BabyAI Unlock from Learning for Long-Horizon Planning via Neuro-Symbolic Abductive Imitation (ABIL). On clean data ABIL reports 99.3% success against 91% for behavioral cloning. We inject wrong-door and backtracking episodes and apply our weighting to ABIL's sub-policies. This tests whether the idea transfers to a hand-written knowledge base.

Metrics are:

  • task success;
  • plan length relative to the clean-data planner;
  • predicate and operator recovery against the clean-data domain;
  • demonstrations needed to reach 80% of clean-data success.

The decisive check comes in three parts, with fumble rates swept to full contamination:

  • Our reward-free method must stay close to NSIL's clean-data success while NSIL-on-noisy-data degrades.
  • It must match or exceed IRIS on Lift and Cans.
  • It must beat length filtering once most trajectories are contaminated.

If NSIL turns out to be robust to fumbles without help, or if length filtering matches us across the sweep, the central premise fails and the paper becomes a negative result.