Judging Demonstrators by What They Could Know: Belief-Relative Imitation of Contingent Behavior

A robot learning from logged human demonstrations has to decide which demonstrations to trust. The usual test is outcome-based: an action counts as good if, given what actually happened, it led toward success. When part of the world is hidden, that test gives the wrong answer. Suppose a driver slows before a blind corner, or a robot detours to check whether a door is open. Judged against the true hidden state, which is known only in hindsight, these actions look wasteful. The door was open, so the detour was a fumble. Meanwhile a demonstrator who barreled through and got lucky looks like an expert. Existing methods that filter out bad demonstrations therefore systematically remove the very behaviors a partially observing agent most needs. These are the hedges, the information-gathering moves, and the branch points of a contingency plan, meaning a plan whose later actions depend on what the agent observes along the way.

We propose to score each demonstrated action against the belief the demonstrator could reasonably have held, rather than against the true state. The belief is a probability distribution over the hidden variables. Scoring against it is only tractable when that belief is small. We therefore restrict attention to the common robotic case where most of the state (pose, velocity, joint angles) is directly measured and only a few discrete hidden modes are uncertain: which door is open, whether an occluded car will yield, which shelf holds the object. In that case the belief is a short probability vector. A critic, meaning a learned estimate of long-run value, can be trained over it offline. Its advantages then separate a sensible hedge from a real mistake. The learned policy shares actions while the mode is ambiguous and splits once the belief resolves, which gives it an explicit contingency structure.

Core Claim

Outcome-conditioned quality estimates are biased against contingent behavior under partial observability. The bias is removable if demonstrator quality is estimated relative to a factored low-dimensional belief. Concretely, let the state be $s=(x,y)$ with $x$ observed and $y$ a discrete hidden mode taking $|Y|$ values. Let $b_y$ be the demonstrator's belief over $y$ given the history up to time $t$. We weight behavioral cloning by

$$w(h_t,a_t)=\exp\big(\beta\,[\,Q(x_t,b_{y,t},a_t)-V(x_t,b_{y,t})\,]\big),$$

where $Q$ is the value of taking $a_t$ from belief-state $(x_t,b_{y,t})$ and then acting well, $V$ is its average under the policy, and $\beta$ is a temperature. In hindsight-conditioned variants, $Q$ and $V$ take the true $y$ in place of $b_y$. Under that substitution the advantage of any information-gathering action is non-positive in expectation, because perfect knowledge of $y$ makes gathering information worthless. That is the bias we target.

This is the exponential advantage weighting of MARWIL (which reweights cloning by estimated advantage and proves monotonic improvement) moved onto the belief of a mixed-observability MDP. Ong et al. factor the state so that belief is maintained only over the hidden part $y$, shrinking its dimension from $|X||Y|-1$ to $|Y|-1$. The new element is not the weighting formula. It is the claim, which we will test, that the input to the critic decides whether contingency behavior survives filtering, and that a low-dimensional explicit belief is what makes a belief-relative critic learnable from small offline datasets.

How It Relates to Prior Work

Three lines of work motivate the gap.

The first is suboptimal-demonstration learning. ILEED learns state-dependent demonstrator expertise as an inner product of state and demonstrator embeddings. Confidence-aware imitation learns per-demonstration confidence weights by bi-level optimization. RelaxDICE replaces exact distribution matching with support regularization so the learner is not tied to low-quality visitation frequencies. All three are posed in fully observed MDPs. Under partial observability they either condition on raw observations, which conflates error with ambiguity, or on true state, which gives the hindsight bias above.

The second is trajectory stitching. IRIS and state-level stitching recombine good fragments of bad demonstrations. Both match fragments by state or observation similarity. Under partial observability that can splice together a segment from a demonstrator who knew the door was open with one who did not. We use belief-state matching instead.

The third is imitative contingency planning. BeTopNet decodes a shared short-term maneuver that branches into long-term trajectories. Contingencies from Observations plans branching closed-loop policies over a learned multi-agent flow. Both supply the branching policy structure we reuse. Both treat their data as correct, either human driving logs or scripted CARLA agents, and neither filters it.

Planning methods constrain the design. Path-tree optimization confines branching to discrete visibility zones over a finite mode set, but it needs an exact simulator and noiseless observations. We keep its discrete-mode structure and drop the model: our belief filter is learned and handles noisy observations. The maximum-likelihood-observation planner keeps belief tractable by assuming the most likely observation, which removes branching altogether. It is a warning that tractability tricks can quietly delete contingency. E-PCA belief compression and approximate factored beliefs are the fallback if the hidden mode set grows beyond a handful of values.

Research Program

The method has three learned pieces, all trained offline.

The first is a belief filter $p_\psi(y\mid h_t)$. We train it by supervised learning on hindsight mode labels. This is the key practical assumption: in most logged robotic and driving data the hidden mode is revealed by the end of the episode. The door was eventually seen, and the occluded car eventually appeared. We need the filter to be calibrated, not just accurate, and will measure calibration directly.

The second is a critic over $(x,b_y,a)$. We train it by fitted-Q iteration with a conservative out-of-distribution penalty, since offline value extrapolation is the known failure mode of MARWIL-style methods.

The third is a policy. It has a shared trunk and $|Y|$ mode-specific heads, mixed by a gate that depends on the belief. This follows BeTopNet's shared-then-branching decoder, but lets the branch point emerge wherever belief sharpens rather than fixing it in time. We will also train a belief-conditioned variant of ILEED expertise, replacing $f_\phi(s)$ with $f_\phi(x,b_y)$. This tests whether the benefit comes from the belief-relative input or from advantage weighting specifically.

The analysis half of the program is a controlled demonstrator taxonomy, which lets us measure bias rather than argue for it. In each environment we generate four kinds of demonstrator. Belief-optimal demonstrators come from SARSOP on the known mixed-observability model. Clairvoyant demonstrators see $y$. Gamblers always commit to the prior mode. Fumblers are belief-optimal demonstrators with injected structured errors. Because the true belief-optimal advantage of every logged action is computable, we can report how well each quality estimator recovers it, separately for hedging actions and for errors.

Baselines

The naive reference is plain behavioral cloning on all data with a GRU policy. We choose the GRU because POPGym, which compared 13 memory models across 15 partially observable tasks, found it the strongest memory model on average. The success-filtered variant clones only successful episodes.

The strongest existing competitor is MARWIL with a recurrent critic over observation histories. A history-conditioned critic is implicitly belief-relative, so it is the baseline most likely to erase our advantage. We also run ILEED, RelaxDICE, and IRIS as the leading representatives of expertise estimation, support regularization, and stitching.

All methods share the same datasets, network trunk sizes, offline gradient-step budget, and a fixed hyperparameter search of equal size, tuned on a held-out demonstrator mix. No method is tuned on the test mix.

Two ablations isolate the key ingredient. Hindsight-MARWIL is identical to ours except that the critic takes the true $y$. Our method without the branching heads isolates the policy architecture.

We found no reported result on Lacuna for any of these methods on partially observable mixed-quality demonstration data. Beating the baselines therefore means the following, measured on our generated benchmarks: statistically higher task success than recurrent-critic MARWIL across five seeds on the high-ambiguity scenarios, together with a hedging rate close to that of the belief-optimal demonstrators.

Why not just…?

Why not just filter to successful episodes? Under partial observability, success filtering keeps lucky gamblers. A demonstrator who ignored an occlusion and happened to face a yielding car succeeds and is retained. A demonstrator who hedged and then hit an unlucky mode may fail and be discarded. The demonstrator taxonomy lets us measure this directly: we predict that success-filtered cloning overweights gamblers in proportion to the prior probability of the favorable mode.

Why not just use a recurrent critic on histories, which already conditions on everything the demonstrator knew? This is the real threat, and in the large-data limit it should match our method. Our bet is that offline, from a few hundred mixed-quality episodes, a critic over a $(|Y|-1)$-dimensional calibrated belief extrapolates less badly than one over raw histories. That bet is plausible because the hindsight-labeled filter gets direct supervision, while the recurrent critic must discover the relevant statistic through bootstrapped value targets. We will sweep dataset size. If the recurrent critic matches ours at every size, the contribution narrows to two things: the diagnosis of hindsight bias, and the branching policy. We will report it that way.

Why not just run ILEED with a memory encoder? ILEED factors skill as state-embedding times demonstrator-embedding. It needs demonstrator IDs and overlapping visitation. Even with a memory encoder, it assigns low expertise to states where a demonstrator's actions vary. Under ambiguity, legitimate hedging varies with the belief, so it looks exactly like inconsistency. The belief-conditioned ILEED variant tests whether giving it the explicit belief fixes this.

Proposed Experiments

The environments are chosen so that the hidden mode is discrete and revealed in hindsight.

The first is a set of door and shelf navigation domains in the style of the path-tree optimization paper, extended with noisy detections. The second is a set of gridworld localization tasks with symmetric rooms. The third is a subset of POPGym tasks whose hidden variable is a small discrete mode. The fourth is a CARLA occluded-intersection scenario like the one in Contingencies from Observations, with our four demonstrator types replacing its scripted agents. For a real-human check, we will collect a small teleoperated set in the door domain, where mode labels come free from the episode end.

We report five metrics. The first two are success and return. The third is hedging rate in states where belief-optimal play gathers information. The fourth is the rank correlation between each method's per-action quality score and the ground-truth belief-optimal advantage. The fifth is AUROC for separating hedges from injected errors.

The decisive check is the comparison with hindsight-MARWIL. If we are right, hindsight-MARWIL scores hedges below errors (AUROC under 0.5 on the hedge-versus-error split), upweights clairvoyant demonstrators, and yields a policy that commits early and fails on unfavorable modes. The belief-relative critic should reverse all three. The dataset-size sweep against the recurrent critic then settles whether the explicit low-dimensional belief carries its own weight. A final stress test grows $|Y|$ from 2 to 16. It measures where the explicit belief stops helping and whether swapping in factored filters recovers the benefit.