Lexicase Weighting for Imitation from Demonstrators Who Disagree

Real demonstration datasets are rarely produced by one optimal expert. A teleoperator who is quick but jerky, a careful one who is slow but gentle, and a novice who fumbles the grasp but places the object cleanly all end up in the same pool. Most imitation methods that relax the optimal-expert assumption still score each demonstration, or each state within it, on a single quality axis. Behavior that departs from the dominant mode is then treated as incompetence. We propose to score each short slice of each demonstration on a vector of generic criteria: progress made, time taken, smoothness, peak contact force. We then decide how much to imitate that slice with lexicase selection, a procedure from evolutionary search that compares candidates one objective at a time in a random order and drops those that fall behind on each objective in turn. The result is a policy trained by weighted imitation and conditioned on a requested trade-off. It needs no hand-written reward, and it keeps "specialist" slices that are excellent on some criteria and poor on others.

Long horizons make this a many-objective problem. If each of $n$ time slices carries $c$ criteria, a demonstration is a point in $M = nc$ dimensions, often twenty or more. At that size Pareto dominance (keeping only solutions that no other solution beats on every objective) marks almost everything as non-dominated, so it gives no selection pressure. A weighted sum cannot reach concave parts of the trade-off surface and must be retuned for every task. Lexicase selection was built for exactly this many-objective regime. Our bet is that it transfers from choosing parents in evolutionary search to choosing which data to imitate.

Core Claim

Weighting imitation by the probability that a demonstration slice survives lexicase selection yields better long-horizon policies from mixed-quality human data than either scalar expertise estimation or scalarized advantage weighting. The weights come from comparing each slice only against slices that start from similar states, under an ordering of objectives biased by a preference vector. The gain should appear in two places: task success on multi-stage tasks, and the range of trade-offs one preference-conditioned policy can reach. It should grow with the number of time slices, which is where Pareto ranking collapses and weighted sums become hardest to tune.

Concretely, let $\xi$ be a demonstration slice and $N(\xi)$ the set of slices whose starting states lie near $\xi$'s in a learned state embedding. Let $\lambda$ be a preference vector over the $M$ objectives, and let $\sigma \sim \pi_\lambda$ be an objective ordering drawn by Plackett–Luce sampling, so objective $i$ is picked next with probability proportional to $\lambda_i$. Then

$$w_\lambda(\xi) = \Pr_{\sigma \sim \pi_\lambda}\big[\xi \text{ survives } \varepsilon\text{-lexicase filtering within } N(\xi) \text{ under order } \sigma\big],$$

where $\varepsilon$-lexicase keeps any candidate within a tolerance $\varepsilon$ of the best on the current objective, which suits continuous scores. The policy $\pi_\theta$ is trained by

$$\max_\theta \; \mathbb{E}_{\lambda}\, \mathbb{E}_{(s,a) \in \xi}\; w_\lambda(\xi)\, \log \pi_\theta(a \mid s, \lambda).$$

The policy only ever reweights real transitions, so it stays on the support of the data. The price is that it cannot exceed the best trade-off some demonstrator actually achieved from a similar state.

Where This Sits in Prior Work

Three lines of work motivate the proposal by showing that ignoring demonstrator quality fails on long, messy tasks. IRIS proposes reachable sub-goals with a generative model, picks among them with a scalar Q-network, and reaches them with a goal-conditioned controller. It lifts Robosuite Lift from 13.7% (behavior cloning) to 81.3%. SPLASH uses human-defined options and whole-trajectory rankings, and shows that low-level cloning wins 2% of capture-the-flag games while options-level cloning wins 99%. Robust Offline Imitation Learning Through State-level Trajectory Stitching mines recovery segments from suboptimal data toward a known expert pool. All three judge quality with one scalar or rely on a verified expert set. We use neither: no reward labels, and no designated experts.

The closest competitors supply our starting point. ILEED learns state-dependent expertise $\rho(s,\omega_i)$ as the inner product of a state embedding and a demonstrator embedding, modeling each action as a mix of the optimal policy and noise. It beats BC, GAIL and IRIS in 21 of 23 settings. Because its expertise is state-dependent, it already captures "good at this stage, bad at that one," which is a serious threat to our proposal. What it cannot represent is systematic disagreement about what "good" means, such as trading speed for gentleness. MARWIL weights cloning by exponentiated advantage and needs a scalar return. RelaxDICE acts as a constraint on our design: it shows that forcing a policy to match the frequency of low-quality data traps it. Our weights address the same problem by zeroing out dominated slices while staying inside the data's support.

The mechanism comes from Objectives Are All You Need. That paper slices an episode into chronological segments, treats each segment as an objective, and uses lexicase selection to beat MAP-Elites on deceptive mazes and Knight's Tour without hand-designed behavior descriptors. It works only with evolutionary search over discrete problems. We move the idea to offline, gradient-trained visuomotor imitation, and we use it to weight data rather than to choose parents. Pareto-set learning methods such as DDPS-MCMC and DOMOO condition one model on a preference vector. They show that uniform preference sampling wastes effort, and that surrogate models invent dominance outside the data. They motivate our choice to never extrapolate in objective space, and we borrow their evaluation language (hypervolume, IGD).

Research Program

The pipeline has four parts, each chosen to avoid per-task reward design.

First, we learn a state embedding with a self-supervised temporal-distance model trained on all demonstrations. It serves two purposes: it defines the neighborhoods $N(\xi)$, and it gives a progress score (predicted steps remaining to the final state). Second, we cut each demonstration into $n$ slices by normalized progress rather than wall-clock time, so slices from demonstrations of different lengths line up. Each slice gets $c$ task-agnostic criteria computed from proprioception and simulator state: progress gained, duration, mean squared jerk, peak end-effector force, and eventual episode success propagated backward. Third, we estimate $w_\lambda$ by Monte Carlo over orderings, which is cheap because neighborhoods hold tens of slices. We add a small floor weight so no state loses all its training data. Fourth, we train a BAKU-style transformer policy (BAKU) that takes $\lambda$ as an extra input. At test time we either fix $\lambda$ to a desired profile or put all its weight on success.

On the analysis side, we will track two quantities as $M$ grows: the fraction of slices that are non-dominated, and the entropy of the lexicase weights. This checks directly the claim that longer horizons push the problem into the regime where lexicase helps and Pareto ranking does not.

Baselines

The naive reference point is plain behavior cloning on all data with the same transformer. We also run its obvious fix: filtering to the top 10% of episodes by length or success, then cloning. These tell us how much any weighting helps.

The strongest scalar-expertise competitor is ILEED, run with its published objective and given the same state embedding and policy backbone as our method. It is the strongest because its state-dependent expertise already covers stage-local skill differences. On Robomimic-style data it is reported to beat BC, GAIL and IRIS in 21 of 23 settings with a 7% average reward gain. The file gives no per-task Square number, so none is claimed here.

The strongest long-horizon competitor is IRIS. On RoboTurk Cans (state-based) it reports 28.3% success against 0% for BC and batch RL. We rerun it on our splits, giving it success-only reward labels, which is more supervision than our method receives.

The scalarized competitor is MARWIL-style exponentiated-advantage cloning on the weighted sum $\sum_i \lambda_i r_i$ of the same criteria. We give it the same set of $\lambda$ values, a matched tuning budget over its temperature, and the same backbone.

The key-ingredient ablations swap only the weighting rule inside our pipeline. One uses Pareto-rank weights (non-dominated sorting within $N(\xi)$). One uses weighted-sum weights. One scores whole trajectories instead of slices, which sets $n=1$. One drops the neighborhoods and compares slices globally. Every method gets the same data, backbone, number of gradient steps, and hyperparameter trials.

Beating the baselines means two things. Our method must exceed ILEED and IRIS in success at the success-favoring $\lambda$. It must also reach a higher hypervolume over (success, duration, jerk, force) than the weighted-sum and Pareto-rank ablations. On multi-stage tasks the margin should widen as $n$ increases.

Why not just…?

Why not just use ILEED, since its expertise already depends on the state? ILEED models a deviation from the dominant action as noise. A demonstrator who is systematically slow and gentle, and consistent about it, gets low expertise wherever the majority is fast, even when gentleness is exactly what a user wants. Whether this matters for success alone, and not just for trade-off coverage, is an open question. The controlled-demonstrator experiment below settles it: if ILEED matches us on success there, our claim reduces to the trade-off-coverage result.

Why not just sweep weights on a scalarized reward and run advantage-weighted cloning? Schrum's work (Constructing Complex NPC Behavior via Multi-Objective Neuroevolution, Objectives Are All You Need) reports that weighted sums wash out specialists and cannot reach concave parts of the front. Separately, the locomotion work in the file (Learning Agile Locomotion via Adversarial Training) reports that hand-tuned weight balances are fragile. With criteria on different scales and $M$ in the tens, the sweep also grows combinatorially. This baseline is a real threat when $M$ is small, and we expect it to tie at $n=1$. The weighted-sum ablation measures exactly where it stops keeping up.

Why not just keep the Pareto non-dominated slices? For $M \ge 4$ the non-dominated fraction approaches one (Constructing Complex NPC Behavior via Multi-Objective Neuroevolution), so this filter degenerates toward plain cloning. Our non-dominated-fraction measurement tests this directly on real demonstration data.

Why not just pick the best slice per stage and stitch, as IRIS or BATS do? Choosing a "best" slice needs a scalar score within each stage, which brings back the weighted-sum problem at a smaller scale. Stitching also needs reward labels. The IRIS baseline, run with more supervision than we use, bounds how much this simpler route gives.

Proposed Experiments and Decisive Checks

The first experiment is controlled. On Robomimic Lift and Square and on D4RL FrankaKitchen (D4RL), we generate data from scripted demonstrators with known, distinct trade-offs: fast and jerky, slow and smooth, force-cautious. Each also gets fumbles injected at stage-specific rates. Since the ground-truth specialists are known, we can check whether the lexicase weights pick them out stage by stage, while ILEED's weights collapse them. The decisive test is this: if ILEED's success matches ours at the success-favoring $\lambda$ and its policy reaches the same trade-off profiles when conditioned, the proposal fails.

The second experiment uses real human data: RoboTurk Cans, Robomimic Square across its proficiency tiers, and Adroit cloned/human. Here we compare against the IRIS and ILEED numbers above. For Adroit pen-cloned we also compare against the SBR score of 95.6 reported in Robust Offline Imitation Learning Through State-level Trajectory Stitching. For FrankaKitchen, no reported result was found on Lacuna for any baseline in this file, so all numbers there come from our reruns.

The third experiment tests long horizons on LIBERO-90, where BAKU reports 90% success. We vary the number of slices $n$ from 1 to 16 and plot success, hypervolume, and weight entropy against $M$. The central prediction is that the gap between lexicase and the Pareto-rank and weighted-sum ablations widens with $n$. If it does not, horizon length is not what drives the benefit, and the many-objective framing is wrong. No hypervolume result was found on Lacuna for any of these manipulation benchmarks, so the hypervolume comparisons will be new measurements, not comparisons to published numbers.