Interaction Duration Silently Rewrites Hand-Written Driving Rewards Across Domains

Hand-written reward functions for driving usually charge a small cost for every tick of elapsed time and a large one-time cost for a crash. The ratio between these two terms quietly fixes how much collision risk the trained agent will accept to avoid waiting. That trade-off depends on how long the agent has to wait. In real traffic, waiting time is set by other drivers: how long an unprotected left turn takes depends on how often oncoming drivers leave gaps, and that varies by city, traffic density and local driving style. A reward tuned to look sensible in one domain can therefore encode a very different risk tolerance in another, even if the policy's perception and representations transfer perfectly. This note proposes to measure this effect and then remove it by construction. Decisions are made over maneuvers whose length is set by interaction events, and the maneuver, not the clock tick, carries an explicit risk budget.

The opportunity matters because the field already knows that hand-written driving rewards often encode absurd preferences. Reward (Mis)design for Autonomous Driving found that 7 of 9 focus papers' rewards mathematically preferred a crash over staying parked. The field also knows that policies degrade across domains. These two failures have been studied separately. The claim here is that part of the cross-domain degradation of reward-trained interactive policies comes from the reward, not the representation. The usual domain-generalization tools, which align features across domains, cannot fix it.

Core Claim

Consider an agent facing a single interaction. It can wait for a gap, which takes a duration $\tau$ (in ticks) set by the other agents. Or it can go now, which risks a collision with probability $p$. With per-tick reward $r_{\text{step}}<0$, crash penalty $C<0$ and discount $\gamma$, the agent is indifferent between the two options at

$$p^{*}(\tau) = \frac{r_{\text{step}}\,(1-\gamma^{\tau})}{(1-\gamma)\,C}.$$

Here $p^{*}$ is the largest collision probability the reward accepts to avoid waiting, so it is the policy's effective risk tolerance. Because $p^{*}$ increases with $\tau$, a domain where interaction partners hold the agent up longer produces a policy that is more willing to take risks, under exactly the same reward. Semi-Markov discounting, the variable-duration form of the Bellman equation where returns are discounted by $\gamma^{\tau}$ rather than a constant (A Deep Value-network Based Approach for Multi-Driver Order Dispatching), makes the same coupling explicit at the decision level.

The proposal makes three claims. First, the distribution of interaction-determined waiting times shifts measurably across driving domains. Second, policies trained with tick-based shaped rewards show a corresponding shift in realized risk per maneuver when moved to a new domain. Third, a design that removes the coupling can reduce this shift without retuning on the target domain. In that design, high-level actions are variable-duration options that terminate on interaction events, such as "gap accepted," "partner committed to yield," or "partner passed," and are trained under a per-option chance constraint on collision rather than a traded-off penalty. An option here is a temporally extended action with its own internal controller and termination rule.

Relation to Prior Work

Several works motivate this gap. Knox et al. (Reward (Mis)design for Autonomous Driving) compute implied preferences and risk indifference points for published rewards. Their analysis is static and in a single domain, and they report that none of the focus papers required the agent to drive more safely than "a legally drunk teenager." We make their indifference point a function of interaction duration and test whether it moves across domains. Domain Generalization for Robust Model-Based Offline Reinforcement Learning (DIMORL) applies risk extrapolation across demonstrator domains. Risk extrapolation is a penalty on the variance of training loss across domains. DIMORL found that better out-of-distribution model likelihood did not reliably translate into better policy return. We read this as evidence that something outside the learned model also governs transfer, and we test whether reward-duration coupling is one such factor.

Other works supply the methods we build on. Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills (HsO-VP) learns driving skills and tests in an unseen CARLA town. Its skills last a fixed 1.0 s, and the authors leave variable-length skills open. Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving (MTHRL-HS) fixes its decision intervals at 1.0 s and 0.1 s by hand. Learning Expensive Coordination: An Event-Based Deep RL Approach (EBPG) lets a leader decide only when discrete events occur, but only in a single-leader particle environment. We reuse EBPG's event-triggered decision idea in a peer-to-peer setting, where termination is caused by another agent's commitment. We also replace the fixed intervals of HsO-VP and MTHRL-HS with event-determined durations.

One line of work bears on whether representation alignment alone suffices. AdapTraj separates domain-invariant from domain-specific interaction features for open-loop trajectory prediction across ETH, UCY, L-CAS, SYI and SDD. It shows that naively pooling domains can hurt. AdapTraj addresses representation shift in prediction. Our question concerns objective shift in closed-loop control, which feature disentanglement leaves untouched.

Research Program

The first stage is measurement. On logged data, we extract interaction episodes such as unprotected turns, merges and unsignalized crossings. For each, we compute the ego agent's waiting duration from arrival to commitment, and we compare these distributions across domains. For real data, we use the six U.S. cities of Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset and the highway traffic in HighD. For simulated data, we use CARLA towns and Highway-Env, varying traffic density and the aggressiveness parameters of the background driver model. Plugging these empirical distributions into the indifference equation gives a predicted shift in risk tolerance for each published-style reward. This stage requires no policy training and can falsify the premise cheaply. If waiting distributions barely differ across domains, the coupling is immaterial, and we would report that.

The second stage tests the prediction with policies. We train flat 10 Hz policies with representative tick-shaped rewards in source domains, using rewards reconstructed from the Knox audit categories. We then measure realized collision probability per interaction in target domains, and we check whether the measured shift tracks the shift predicted by the equation.

The third stage builds the remedy, a constrained semi-Markov policy. A high-level policy chooses among maneuver options: wait, creep, commit, follow, or change lane. Each option terminates when a detected interaction event occurs or a timeout is reached, so its duration is variable and partly caused by partners. Low-level tracking runs at 10 Hz, as in MTHRL-HS. The objective is efficiency, an explicit value of time stated in seconds. Safety enters as a per-option chance constraint, $P(\text{collision}\mid s,\omega)\le\delta$, where $\omega$ is the option and $\delta$ is a designer-chosen risk budget per maneuver. The constraint is enforced with a Lagrange multiplier learned per option type. Because the constraint is attached to a maneuver rather than accumulated time, the risk tolerance is $\delta$ whatever the wait. Asynchrony between agents is handled because only the ego's decision points are events. Partners act continuously in the simulator, and their reactions are absorbed into the option's realized duration and outcome. This sidesteps the joint-termination problem rather than solving it, and we say so.

Baselines

The naive reference point is a flat 10 Hz policy trained with a tick-shaped reward of the kind Knox et al. audited, tuned once on source domains and deployed unchanged. This baseline predicts the failure, and every other method is compared against its shift in risk across domains.

The strongest existing method on this question is the HsO-VP hierarchical skill planner, which reports a Driving Score 6.4% higher than IRIS on unseen Town05 (HsO-VP). It is strongest because it is the only cited method that combines temporal abstraction with an explicit unseen-town evaluation. No absolute Driving Score for HsO-VP was found on Lacuna, so we will rerun it on the same CARLA split: train on Town01, 03, 04 and 06, test on Town05. It will use the same offline data, the same number of gradient steps and the same hyperparameter search budget as our method. Beating it means two things. Our method should keep per-maneuver collision probability in Town05 within a factor of two of its source-domain value, while HsO-VP's moves more. And it should do this at equal or better Driving Score.

On Highway-Env and HighD, the reference is MTHRL-HS. It reports a 0.03% collision rate in Highway-Env and 0.02% on HighD (MTHRL-HS), both in the training domain. We will rerun it under our density and aggressiveness shifts. The question is whether its collision rate stays near its reported values when partner behavior changes.

As a representation-level domain-generalization baseline, we apply DIMORL-style risk extrapolation across source domains to the flat policy's model training. This tests whether aligning across domains alone removes the risk shift.

Two ablations isolate the key ingredient. The first keeps the variable-duration event options but returns to a shaped per-tick reward. This tests whether options alone help. The second keeps the per-maneuver chance constraint on a flat 10 Hz policy, with maneuvers segmented post hoc. This tests whether the constraint alone suffices. All methods share data, simulator seeds and tuning budget, with tuning done on source domains only.

Why not just…?

Why not just retune the reward on each target domain? Retuning would remove the shift if target-domain rollouts were available. However, it gives up domain generalization by definition, and it is exactly the trial-and-error design Knox et al. flag as a failure mode. We include a target-retuned flat policy as an oracle upper reference. If our method approaches the oracle's risk stability without target access, the retuning argument loses force. If it does not, we will report the size of the gap.

Why not just imitate human drivers and skip reward design, as RAIL does? Randomized Adversarial Imitation Learning for Autonomous Driving replaces the reward with a discriminator. Its stacked policy reaches 70.38 km/h against a 68.83 km/h expert average, but only in a single Unity highway environment. Imitation inherits the source demonstrators' risk tolerance and is bounded by their quality. Whether that tolerance adapts sensibly to a new domain is an open empirical question. This is a real threat. We include behavioral cloning and an adversarial imitation baseline, measure their per-maneuver risk shift alongside ours, and will say so if they are as stable.

Why not just add an episode-level safety constraint to a flat policy? A constraint on per-episode collision probability is also insensitive to waiting time, so it may capture most of the benefit without options. This is the most serious alternative. The second ablation tests it directly. Our expectation that per-maneuver constraints transfer better rests on the number of interactions per episode varying across domains, which the measurement stage quantifies. If the flat constrained policy matches ours, the contribution narrows to the diagnosis plus the constraint, and options would be justified only by credit-assignment efficiency.

Why not just use an average-reward formulation and drop discounting? Average-reward optimality, argued for ongoing processes in Process-Oriented Planning and Average-Reward Optimality, removes the $\gamma^{\tau}$ factor. It does not remove the accumulated cost of time, so the indifference point still grows with waiting. We include an average-reward flat baseline to confirm this empirically.

Proposed Experiments and Decisive Checks

The first decisive check is the measurement study. Interaction waiting distributions are compared across WOMD cities, CARLA towns, Highway-Env density settings and HighD. We report effect sizes and the predicted change in $p^{*}$ for each audited reward style.

The second check is predictive. Across source-to-target shifts, the realized change in per-interaction collision probability of tick-reward policies should correlate with the change predicted by the indifference equation. Weak correlation would mean the coupling is real on paper but swamped in practice, and that is a publishable negative result.

The third check is the remedy. On CARLA Town05 and on Highway-Env with shifted partner aggressiveness, we report per-maneuver collision probability relative to the budget $\delta$, its ratio between target and source, Driving Score and route completion in CARLA, and mean wait time. We also run automated versions of Knox's preference checks on each trained policy's realized behavior.

The method succeeds if its target-to-source risk ratio stays near one while the flat shaped baseline, HsO-VP and the risk-extrapolation baseline drift. It must also hold Driving Score at least level with HsO-VP. Finally, the two ablations together must show which ingredient carries the effect.