When a robot learns by copying human demonstrations that are partly bad, a common fix is to weight each demonstrated action by how "expert" it looks. Low-quality actions are pushed down, and the policy fits a sharpened version of the data. The sharpness is set by one number, an inverse temperature $\beta$. At $\beta=0$ every action counts equally, which is plain behavioral cloning. As $\beta$ grows, the method approaches hard filtering. Practitioners treat $\beta$ as a statement about what to imitate. This proposal argues it is just as much a statement about how well gradient descent can reach it. Sharper weights make each minibatch dominated by a few samples, so stochastic gradients get noisier. Standard stochastic-approximation theory then says that a constant learning rate $\alpha$ leaves the policy bouncing inside an error ball whose size grows with both $\alpha$ and that noise. So the temperature that best matches the expert is often one the optimizer cannot settle into.
The idea is to measure this cost directly and schedule around it. The quantity that drives the error ball is the spread of the weights inside each minibatch, which costs almost nothing to compute. We use it to raise $\beta$ gradually as the learning rate shrinks. Early training runs at low temperature, uses almost all of the data, and moves quickly toward the shared structure of the behavior. Late training runs at high temperature with small steps, so it only has to close a short remaining gap. The setting is strictly offline and mostly reward-free, which is exactly where imitation from suboptimal data is used.
Core Claim
We read weighted imitation through control as inference. That approach recasts decision-making as conditioning on a binary "this behavior is optimal" event $O$. As in the pMDP-MO construction of TRGAIL, we take the likelihood of that event from a learned discriminator $d(s,a)$, a classifier that scores how expert a state-action pair looks, and define $P(O=1\mid s,a)\propto (d/(1-d))^{\beta}$. The policy update is then the M-step of an EM loop: a supervised fit to the posterior over actions given $O=1$. The weights are
$$w_\beta(s,a)=\left(\frac{d(s,a)}{1-d(s,a)}\right)^{\beta}.$$
At $\beta=1$ these are the density-ratio weights used by discriminator-weighted cloning. As $\beta\to\infty$ they become a hard filter. When rewards are available, the analogue is MARWIL's $\exp(\beta\hat A)$ (Qing Wang et al.). Our central claim is that the gap between the trained policy and the expert decomposes, to first order, as
$$\mathcal{E}_t \;\lesssim\; B(\beta) \;+\; c\,\alpha\,\kappa(\beta) \;+\; (1-\alpha\mu)^t D_0, \qquad \kappa(\beta)=\frac{\mathbb{E}[w_\beta^2]}{\mathbb{E}[w_\beta]^2}.$$
The symbols are as follows. $B(\beta)$ is the bias of the tempered target relative to the expert, which shrinks as $\beta$ grows. $\kappa(\beta)$ is the inverse effective-sample-size ratio of the weights, which grows roughly exponentially in $\beta$ when the log-odds are spread out. $\mu$ is a local curvature constant, $D_0$ is the initial distance to the target, and $c$ is a constant. The middle term is the constant-step error ball. It has the same structure as the $\alpha$-proportional neighborhoods proven for Adam-type optimizers by Iiduka and the $\alpha$-dependent stationary spread of constant-step temporal-difference learning in Amortila et al., with the weight-variance ratio multiplying the gradient noise.
Two consequences follow, and both can be tested. First, for any fixed learning rate and budget there is an interior best $\beta$, and it is predictable from $\kappa$. Second, a schedule that keeps $\alpha_t\,\kappa(\beta_t)$ roughly constant while $\alpha_t$ decays reaches high-$\beta$ accuracy in fewer steps than any fixed pair. The reason is that most of the distance $D_0$ is traveled while $\kappa$ is small.
Research Program
Theory. The first part is analysis in a setting where the claim can be proven: a linear-softmax policy over tabular states, trained by weighted SGD. Following the Markov-chain view of Amortila et al., constant-step weighted SGD forms a time-homogeneous chain. We will bound its stationary spread in terms of $\alpha\kappa(\beta)$. We will then show that a continuation schedule, which warm-starts each temperature stage from the previous stage's solution, attains a target error with fewer iterations than direct training at the final $\beta$. The argument is the standard homotopy one: each stage contracts geometrically over a short distance.
The discriminator is learned jointly, so the weights drift over training. This is exactly the non-stationarity the file identifies as unanalyzed (Gap 3). We will handle it only in the two-timescale case, with the discriminator frozen or updated much more slowly than the policy, and say so plainly.
Algorithm. The second part is a practical schedule, which we call ESS-matched tempering. It has three steps. We first train the discriminator with positive-unlabeled learning on a small expert anchor set plus the unlabeled mixed data, exactly as in DWBC. We then fix a learning-rate decay. At each step, we choose the largest $\beta_t$ whose minibatch $\kappa$ keeps $\alpha_t\kappa(\beta_t)$ below a single threshold.
This borrows the adaptive-tempering rule from sequential Monte Carlo, the family of samplers that move through a sequence of sharpening targets. It also recalls the cost-annealing that VGPS used heuristically when refining a flawed demonstration into a walking gait. What we add is a stopping rule tied to the optimizer's error floor rather than a hand-set temperature path.
The whole method needs one extra threshold hyperparameter. It needs no Bellman backups and no environment interaction.
Benchmarks. Evaluation runs on the D4RL locomotion, AntMaze and Adroit suites (Fu et al.). We use the expert-plus-random and expert-plus-medium mixtures that DWBC and RelaxDICE use, with the expert fraction swept from 5% to 50%. We also use the robomimic multi-human "Worse/Okay/Better" splits (robomimic), where suboptimality is structured human error rather than injected noise.
Baselines
Naive reference. Plain behavioral cloning ($\beta=0$) with the same network, optimizer and cosine decay. This shows how much any weighting helps. ILEED (Imitation Learning by Estimating Expertise of Demonstrators) reports that plain cloning collapses when one in ten demonstrators is expert, so we expect large gaps at low expert fractions.
Closest method. DWBC (paper) trains the same positive-unlabeled discriminator and the same kind of weighted cloning loss, but at a fixed weighting and a constant learning rate. It is the most direct competitor because our method changes only the schedule. DWBC reports winning 27 of 36 D4RL and Adroit tasks against ORIL, BCND and DemoDICE. We found no per-task normalized score for DWBC on Lacuna. We will rerun it from the same discriminator checkpoints. Beating it means a higher mean normalized score across the mixture sweep, at equal gradient steps, outside seed-level confidence intervals.
Strongest reported method. ReDICE (paper) is a distribution-correction method. It replaces minimax training with Gumbel regression and uses temperature-scaled policy extraction. It reports an average normalized D4RL return of 97.68, against 81.49 for ReCOIL with temperature scaling. That average is over ReDICE's own task set, and ReDICE uses value estimation, so we will rerun it on our exact mixtures under the same step budget. We expect it to win on stitching-heavy AntMaze tasks, where pure weighted cloning is structurally limited. The fair target there is to close most of the gap at a fraction of the compute. Beating ReDICE outright is only claimed for locomotion and robomimic.
Tuned fixed pair (the real test). The strongest in-family competitor is the best fixed $(\beta,\alpha)$ pair with cosine decay. It gets the same number of configurations as our method gets for its single threshold, times the size of a small $\beta$ grid. The comparison is fair only if we also report the result when the fixed-pair baseline is allowed the full grid. That is the regime where a practitioner would actually be.
Ablations. Four ablations isolate the key ingredient. We schedule $\beta$ with a constant learning rate. We decay the learning rate with a constant $\beta$. We warm-start with a single jump from low to high $\beta$. We follow a linear $\beta$ ramp of the same length that ignores $\kappa$. If the linear ramp matches the ESS-matched schedule, then the measured-variance rule is not what matters.
Why not just…?
Why not just grid-search $\beta$ with a standard learning-rate decay? This is the most serious threat. If the fixed-pair baseline with a full grid matches the schedule, the second part of the claim fails. What survives in that case is weaker but still useful: that $\kappa$ predicts the grid's winner, so the grid can be replaced by one cheap diagnostic. The experiments measure both outcomes. We expect a real gap only at low expert fractions, where the best $\beta$ is large and $\kappa$ explodes, so that direct training at that $\beta$ needs a very small learning rate over the whole run.
Why not just filter the top trajectories and clone them? Hard filtering is the $\beta\to\infty$ endpoint, so our decomposition predicts it has the largest variance term and no low-temperature warm-up. RelaxDICE reports that strict matching and hard filtering fall into a "pessimism trap" and collapse on noise-dominated splits, where it beats DemoDICE by more than 30%. D4RL documents that cloning cannot stitch sub-trajectories. We include top-k filtering as a baseline, run at each expert fraction.
Why not just clip or self-normalize the weights? Clipping bounds $\kappa$ directly and is a genuine alternative. In our terms it trades variance for extra bias, because it silently changes the target. We run clipped and per-batch self-normalized weights as baselines, each with a tuned clip level. If clipping at a tuned level matches ESS-matched tempering, we will report that the two are equivalent ways of controlling the same term.
Why not just run offline RL on rewards? Our main setting has no rewards. TRAIL (paper) reports that reward-free imitation can match offline RL with full rewards on DeepMind Control. DWBC reports roughly 7× faster training than value-iteration-based imitation. On reward-labeled D4RL splits we add the MARWIL-style advantage variant, and use ReDICE as the value-based reference.
Proposed Experiments
Check 1: does the predicted error floor appear? On a grid of $(\alpha,\beta)$ we will measure late-training jitter, defined as the variance of the evaluation score across the last twenty checkpoints within a seed. We will test whether it scales linearly with $\alpha\kappa(\beta)$ across tasks. If jitter tracks $\alpha$ but not $\kappa$, the mechanism is wrong.
Check 2: equal budgets. Schedule versus tuned fixed pair at equal gradient steps and equal configuration counts. We report normalized score and steps to reach 90% of the best fixed-pair score.
Check 3: dependence on data quality. The schedule's advantage should grow as the expert fraction falls from 50% to 5%. It should be near zero when data are mostly expert, because there the best $\beta$ is small and $\kappa$ stays near one. Seeing the advantage without this dose-response pattern would suggest a tuning artifact.
Check 4: structured human errors. Robomimic's graded human splits test whether a discriminator-derived optimality likelihood still gives useful weights when errors are systematic rather than noise-like. The file flags this as the weakness of ILEED-style noise models.
Stated risk. Low-temperature warm-up with a unimodal policy head could land in an average of two modes and then refine the wrong one. We therefore repeat Checks 2 and 3 with a mixture-of-Gaussians head, and report mode-collapse rates on the multimodal robomimic tasks.
References
- Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model
- Exponentially Weighted Imitation Learning for Batched Historical Data
- Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks
- A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms
- Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
- Variational Policy Search via Trajectory Optimization
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning.
- Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Imitation Learning by Estimating Expertise of Demonstrators
- Revisiting Distribution Correction Estimation for Offline Imitation Learning with Suboptimal Dataset
- TRAIL: Near-Optimal Imitation Learning with Suboptimal Data