When a learned driving policy is trained by imitation, its small mistakes carry it into situations the human demonstrations never covered, and the errors grow from there. The standard fix is interactive imitation. You let the student drive, and whenever it drifts you ask an expert what it should have done, then add that answer to the training data. In traffic this creates a problem that single-agent methods never face. The student has to drive among other cars while it is still bad, and those cars react to its mistakes. A hesitant merge invites a cut-in. An aggressive nudge can be answered with a refusal to yield. Current systems decide when to take control away from the student either by asking "is this state unfamiliar?" or by predicting the other cars as if they would continue on their current paths. Both questions miss the danger that comes specifically from how others respond to what the student does.
We propose to decide when the expert takes over, and which states get labeled, using a reaction-aware budget check. At each step we ask a population of models of the other drivers, some courteous, some indifferent, some unyielding, how they would respond to the student's intended action. We then check whether the worst-case cost of that response, added to the cost already spent in the episode, would exceed a fixed safety budget. The claim is that this check matters most when the other drivers are reactive and not polite. There it should give fewer crashes during data collection at the same number of expert queries, and a final policy that holds up better against held-out driver types, compared with novelty-triggered or constant-velocity-supervised interactive imitation. When the other drivers are merely replayed from logs, it should give no advantage. We treat that prediction as a falsification test.
Core Claim
Covariate shift in multi-agent imitation has two parts. The first is the familiar one: the student's own states drift away from the demonstrations. The second is that other agents' behavior changes because the student is driving, so the student sees responses that never appear in the logs. Treating others as fixed background, which Wen et al. call "level-0" reasoning and show leads to cyclic or poor learning dynamics, hides this second part from both the learner and the safety mechanism.
Our contribution is a handover rule for interactive imitation. It is a constraint check in the style of a constrained Markov decision process, meaning a formulation where the agent must keep expected accumulated cost under a budget. The cost-to-go in the check is evaluated under reactive opponent models conditioned on the student's own action. Let $C_{0:t}$ be the collision/off-road cost accumulated so far in the episode, $B$ the episode budget, $\mathcal{K}$ a finite set of reactive models of the other agents, and $\hat{C}_k(s_t, a)$ the predicted future cost when the ego takes action $a$ in state $s_t$ and the other agents respond according to model $k$. The student keeps control only if
$$C_{0:t} + \max_{k \in \mathcal{K}} \hat{C}_k(s_t, a^{\text{student}}_t) \le B,$$
and otherwise the expert drives for a minimum window.
This is the backward-value-function bound of Satija et al.. They sum cost-so-far and expected future cost to turn an episode budget into a per-state check. We keep that construction but evaluate the future-cost term under a set of reactive opponents instead of a single stationary environment. A second quantity, the reaction gap, is the spread in predicted opponent responses to the student's action versus the expert's action. We use it to prioritize which handed-over states the expert labels.
Research Program
The system has three learned pieces and one rule. The first piece is the student, a tokenized trajectory policy pretrained by behavior cloning on the Waymo Open Motion Dataset. The second is the opponent population. It contains log replay, the Intelligent Driver Model (a rule-based car-following model), learned closed-loop sim agents fine-tuned in the manner of Peng et al., and deliberately non-courteous variants, such as reduced-yielding driver-model agents and agents fine-tuned to close gaps. Diversity is encouraged with the population and trajectory-diversity ideas of Jing et al.. The third piece is a set of per-model cost critics $\hat{C}_k$, trained on rollouts in which the ego takes perturbed actions so the critics see ego-conditioned responses.
The expert is a privileged optimization-based planner with access to the simulator state. This is a real limitation, since it is not a human. It is also what makes on-policy relabeling possible at the scale of the Waymax simulator, Waymax, which runs batched closed-loop rollouts at over 5000 Hz on one GPU.
The loop is dataset aggregation. The student drives, the budget check hands control to the expert when violated, and the expert holds control for a minimum window. That hysteresis is borrowed from RND-DAgger. Expert labels, weighted by reaction gap, are added to the data before retraining.
The population is not a formality. Waymax reports that learners trained against polite rule-based agents learn aggressive cut-ins that fail against unyielding drivers. A check built only from courteous models would approve exactly those cut-ins. Including unyielding models is what makes the worst case over the population meaningful.
Any guarantee is only relative to $\mathcal{K}$. The open problem noted for population training, vulnerability to "wild" out-of-distribution opponents, applies directly here. We therefore evaluate against driver types held out of the population and report how much the gains shrink.
The work supplies methods from three places. Satija et al. supply the budget decomposition. RND-DAgger supplies the handover mechanics. Jing et al. supply population diversity.
Several prior works motivate it. Tu et al. show that stability constraints can remove the horizon dependence of imitation error. They assume known single-agent dynamics, which is exactly what reactive opponents break. Rajaraman et al. formalize the quadratic-in-horizon loss of offline cloning that makes interactive correction necessary.
Adaptive probabilistic shielding constrains the design. It shows that worst-case checks under high uncertainty block the exploration needed to learn. The max over $\mathcal{K}$ risks the same conservativeness trap, and we measure it directly as the fraction of steps handed over.
Baselines
The naive reference is behavior cloning on WOMD with no interaction, evaluated closed-loop in Waymax. It fixes the floor for final collision rate. It has zero training-time risk because it never drives during training.
The first interactive baseline is RND-DAgger Biré et al.. It hands over when the prediction error of a network distilled from a fixed random target spikes, signalling an unfamiliar ego state, and it holds control for a minimum expert time. It is the strongest published way to cut expert effort in interactive imitation. The paper reports reaching peak performance three times faster in expert time than Ensemble-DAgger in its 3D Maze task. It is single-agent, so no reported result was found on Lacuna for RND-DAgger in multi-agent driving.
We run it with the same student, expert, simulator, opponent population during rollouts, and minimum window. Its threshold is swept so that its expert-query fraction matches ours at several levels. We beat it if, at matched query fraction, we record fewer training-time collisions against reactive and unyielding opponents and a lower final collision rate on held-out driver types.
The second baseline is a supervisor that overrides the student with a planner using constant-velocity predictions of other cars, following Lubars et al.. They report a 0% crash rate in SUMO merging, including in out-of-distribution slow traffic where pure DDPG crashed 85.1% of the time. Those opponents were Krauss rule-based drivers. That numerical target is the one we must match, and the question is whether it survives unyielding reactive opponents. We port the supervisor into the same aggregation loop and match its override rate to ours. We also add a replication on the SUMO on-ramp merge with Krauss traffic, where we expect parity and not improvement.
The third baseline is closed-loop fine-tuning with a collision penalty, as in Peng et al.. They report WOSAC collision score rising from 0.673 for pretrained cloning to 0.877 after fine-tuning. That result is for sim agents, not an ego planner under a budget, so it bounds what penalty fine-tuning achieves on the realism side rather than being directly comparable. We add a Lagrangian variant, a learned penalty weight in the style of PPO-Lag. For PPO-Lag, Safety Gymnasium reports a 98% reduction in safety violations with a 45% reward drop on velocity tasks. Both variants get the same environment-step budget and tuning sweeps.
Two ablations isolate the key ingredient. In the "static-check" ablation, the $\hat{C}_k$ are replaced by critics trained with opponents that do not react to the ego, the ForSim-style level-0 predictor noted in ForSim. In the "no-reaction-gap weighting" ablation, the check is kept but all handed-over states are labeled uniformly. If the static-check ablation matches the full method, the central claim is false.
Why not just…?
Why not just use a state-novelty trigger like RND-DAgger? Novelty measures how unfamiliar the ego's own situation is. It cannot tell a familiar-looking gap that an unyielding driver will close from one a polite driver will open. The RND-DAgger authors note that the detector tracks only ego features and cannot separate environmental novelty from strategic change. This is a genuine competitor in scenes with few interactions, and the matched-query comparison, split by opponent type, will show where each wins.
Why not just fine-tune with a collision penalty, or a Lagrangian? Penalty fine-tuning controls the final policy but says nothing about crashes during data collection. Peng et al. and Li et al. both document the collapse into freezing when the penalty is high. A Lagrangian removes the manual tuning but still enforces the constraint only on average and asymptotically. We report training-time violations explicitly, which is where we expect to win. If a Lagrangian matches us on final held-out collision rate and progress, we will say so.
Why not just train against a diverse opponent population and skip explicit reaction modeling? This is the strongest threat, because diversity alone may produce a student that is robust enough. The RND-DAgger baseline and all fine-tuning baselines therefore roll out against the same population. The static-check ablation then keeps the population but removes ego-conditioned reactions from the check. Any gain that survives both comparisons is attributable to reaction-aware handover rather than population exposure.
Why not just use a supervisor with constant-velocity predictions? It worked perfectly against Krauss drivers (Lubars et al.). The same reliance on politeness is what Waymax warns against. The unyielding held-out opponents test exactly this.
Proposed Experiments
The main benchmark is Waymax on WOMD validation scenarios. We select interaction-heavy subsets such as merges, unprotected turns, and lane changes, and evaluate on all scenarios. The primary metrics come in three groups.
The first group measures training risk and cost:
- training-time collisions and off-road events per thousand student rollouts;
- expert-control fraction;
- the fraction of steps blocked by the check, which measures conservativeness.
The second group covers the final policy's closed-loop performance:
- collision rate;
- route progress;
- off-road rate.
These are reported per opponent type in a leave-one-type-out protocol. In that protocol, one family, for example gap-closing learned agents, is excluded from $\mathcal{K}$ and from training rollouts and used only at test.
The secondary benchmark is the SUMO on-ramp merge. It checks that we recover the 0% crash rate of Lubars et al. when opponents are polite rule-based drivers.
Three results would be decisive. The method's advantage over RND-DAgger and the static-check ablation should be near zero with log-replay opponents and grow with opponent reactivity and non-courtesy. If the advantage is uniform across opponent types, it comes from something other than reaction modeling. The advantage on held-out opponent families should be smaller than on in-population families but still positive; if it vanishes, the method only memorizes its population. The handover fraction should stay low enough that the student still gathers on-policy data, and we report the curve of reward against training-time violations to check that the worst-case check has not fallen into the conservativeness trap described by adaptive probabilistic shielding.
References
- Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving
- An Open-Ended Learning Framework for Opponent Modeling
- Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research
- Efficient Active Imitation Learning with Random Network Distillation
- Closing the Closed-Loop Distribution Shift in Safe Imitation Learning
- Toward the Fundamental Limits of Imitation Learning
- Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning
- Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- ForSim: Stepwise Forward Simulation for Traffic Policy Fine-Tuning
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning