Friction-Aware Trajectory Forecasting Learned from Randomized Physics

Self-driving systems predict where nearby cars will go, usually as several possible futures: turn left, go straight, brake. Today's predictors learn these futures from logs of ordinary driving on dry roads. On dry roads cars never slide, so the models never see what a car physically cannot do on ice or wet asphalt. On a slippery road such a predictor will still say the car ahead can take a tight turn at speed. The planner then trusts a maneuver the car cannot perform, and it never considers the slide the car is actually about to make.

We propose to train a forecaster entirely on simulated traffic in which road grip, vehicle mass and tire stiffness are randomized from scene to scene. This technique is called domain randomization: deliberately varying the physics of a simulator so that a learned model cannot overfit to any one setting. The forecaster has two parts. The first infers a single grip estimate for the whole scene by pooling the recent motion of every visible car, since all cars on the same patch of road share it. The second generates futures by rolling a tire-based vehicle model forward under that estimated grip. The claim is that a predictor built this way transfers without fine-tuning to physics it never trained on. On low-grip roads, it should both stop proposing impossible futures and start proposing the slides and understeer that really happen. On ordinary dry-road data it should cost almost nothing in accuracy.

What exists and what is missing

The closest work is Trajectron++. It predicts multiple futures with a conditional variational autoencoder, which is a generator that samples futures from a learned latent "intent" variable. Instead of predicting positions directly, it predicts accelerations and steering rates and integrates them through a unicycle model, a purely geometric description of a car's motion. That guarantees smooth paths, but it does not guarantee physically achievable ones. A unicycle can turn at any speed, because it has no tires, no slip and no grip limit. Second-Order Graph ODEs goes further toward physics: it decodes trajectories as Newtonian accelerations and cuts collision rates under sudden obstacles from 16.8% (AgentFormer) to 7.0%. Its "forces", however, are abstract latent vectors with no tie to tire grip, and it is trained only on fixed datasets.

Two other lines of work motivate the proposal. Control papers show that geometric car models break down at speed because they omit lateral tire forces (Alcalá et al.). They also show that grip limits drift with wear and rain (Kalaria et al.). Kalaria et al. handle this drift with fast online residual learning, but only for the ego car and with no forecasting of other cars. Diagnostic work shows that strong learned predictors lean on map cues rather than motion physics: AgentFormer's final displacement error rises from 3.89m to 4.21m without maps and to 5.41m on curves (Beyond ADE and FDE).

Two further results supply the method. Dynamics randomization yields zero-shot transfer for low-level controllers (Tan et al.). Exarchos et al. report that randomizing body geometry beat randomizing dynamics for quadrupeds (average reward 6215 vs 3099). That result is a direct challenge to our choice of randomizing grip, and we test it rather than assume it away.

The missing piece is a multi-agent, multi-future predictor that is trained with randomized physics and conditions its futures on a scene-level physical quantity inferred from what it observes. None of the papers above does this.

Core Claim

Observed trajectories carry information about road grip. A predictor trained across randomized grip can read that information from the histories of all agents in a scene and adjust which futures it considers possible. Concretely, the decoder integrates a dynamic bicycle model, a two-axle car model whose tire forces follow a saturating curve. The model is governed by a friction coefficient $\mu$ that limits total tire acceleration:

$$ \sqrt{a_{\text{lon}}^2 + a_{\text{lat}}^2} \le \mu g, $$

where $a_{\text{lon}}$ and $a_{\text{lat}}$ are the longitudinal and lateral accelerations and $g$ is gravity. Rather than fixing $\mu$, the model infers a posterior over a scene-shared grip $\hat\mu$ from all agents' histories $h_{1:N}$. It then samples intents $z$ conditioned on that estimate:

$$ p(\tau_{1:N} \mid h_{1:N}) = \int \sum_{z} p(\tau_{1:N} \mid z, \hat\mu)\, p(z \mid h_{1:N}, \hat\mu)\, q(\hat\mu \mid h_{1:N})\, d\hat\mu . $$

Here $\tau_{1:N}$ are the future trajectories. Because the mode prior depends on $\hat\mu$, low grip does more than shrink each future. It removes some modes, such as a fast tight turn, and adds others, such as an understeer drift toward the outside of a curve.

Research Program

We will build a lightweight multi-vehicle simulator with dynamic-bicycle cars and saturating tire curves. Background cars are driven by model-predictive controllers with sampled intents (lane choice, turn, yield, emergency brake). Per scene we randomize grip (roughly ice to dry asphalt), mass, cornering stiffness, actuation latency and, for the geometry ablation, wheelbase and track width. The randomized ranges are a research variable in their own right. The literature warns that over-wide randomization produces sluggish, conservative behavior, so we will compare uniform ranges against ranges matched to plausible road conditions.

The predictor reuses the Trajectron++ graph encoder and discrete-intent generator. We swap its unicycle for the dynamic bicycle and add a grip head that pools all agents' histories. Training is entirely in simulation, with the grip head supervised by the simulator's true $\mu$. This uses the privileged-information idea from Asymmetric Actor Critic: the simulator's true value is a training signal only and is unavailable at test time.

Transfer is evaluated with no fine-tuning on three targets. The first is CARLA multi-agent scenes with altered per-wheel friction, a different physics engine from training (CARLA). The second is the Assetto Corsa human laps near the grip limit, which come from a professional-grade tire model and real human drivers (racing benchmark). The third is nuScenes, used as a check that nominal real-world accuracy holds (nuScenes).

This plan has an honest limitation. The file contains no real-world, multi-agent, low-grip dataset, so "real" low-grip transfer is approximated by cross-engine and human-driver targets. Logging low-grip real traffic is the natural follow-up.

Baselines

The naive reference point is a constant-velocity extrapolation and an off-the-shelf Trajectron++ trained on nominal data. For Trajectron++, the file gives a nuScenes vehicle ADE at K=10 of 1.51m, as reported in the AgentFormer paper. These baselines show how much a low-grip shift hurts a predictor that knows nothing about physics.

The strongest existing data-driven method is AgentFormer, which reports nuScenes vehicle ADE at K=10 of 1.45m. That is the best number in the file. We also include Second-Order Graph ODEs as the strongest physics-flavored competitor, given its 7.0% sudden-obstacle collision rate. Each competitor is run two ways: on its original real data, and retrained on exactly our randomized simulated corpus with the same number of gradient steps and the same hyperparameter-search budget. The second version separates the value of our architecture from the value of the randomized data itself. Beating them means the following. On nuScenes we stay within about 0.05m of 1.45m ADE. On the low-grip targets we achieve lower minimum final displacement error, lower negative log-likelihood and a lower rate of grip-violating samples. We also cover more of the observed slide events.

Three ablations isolate the key ingredients. The first is the same model with grip fixed at the dry value, which tests whether inference matters. The second replaces scene-pooled grip with a per-agent estimate, which tests whether sharing grip across cars matters. The third replaces the dynamic bicycle with a unicycle clamped by the grip inequality using the inferred $\hat\mu$, which tests whether tire dynamics matter beyond the limit itself. An oracle-$\mu$ variant gives the upper bound.

Why not just…?

Why not just train AgentFormer on more data? Real logs almost never contain sliding cars, so more data adds more of the same dry-road physics. The file shows such models already ignore physics when map cues are removed (FDE 5.41m on curves, per Beyond ADE and FDE). The fairer version of this alternative is AgentFormer trained on our randomized simulated data, which we run directly. If it matches our model, the contribution is the data, not the physics layer, and we will report it that way.

Why not keep the kinematic model and let the ego planner handle dynamics? The ego controller can correct only its own slip. If the forecast says another car can make a turn it cannot, the planner works from false premises; Alcalá et al. shows the kinematic model's error grows exactly in this regime. We test this alternative by running the same downstream planner in CARLA on top of each predictor and comparing collisions and near-misses.

Why not just clamp a kinematic model's acceleration at the grip limit? This is the real threat. Within the limit, a clamped unicycle may match a tire model, and the clamp is far simpler. It cannot, however, represent heading diverging from velocity, which is what happens in understeer and drift. The clamp ablation will settle the question. If the gap appears only in slide events and is small, we will adopt the clamp and present the scene-pooled grip inference as the contribution.

Why not estimate grip online with system identification instead of randomizing? Online identification needs post-change error before it converges. In Kalaria et al., the non-adaptive controller spent 4.32s violating track limits versus 0.46s for the adaptive one, which still means some violation while adapting. Gaussian-process versions take 0.25–0.3s per update. We include an extreme-learning-machine residual estimator fit to other agents' observed motion as a baseline. If it equals our inferred grip after a grip step change, randomized training adds nothing over identification.

Proposed Experiments

The decisive check is a controlled grip step. In CARLA scenes, grip drops partway through the history window. We measure how quickly each model's predicted futures adjust, using the grip-violation rate of its samples and the minimum final displacement error over the following three seconds. Our prediction is that scene pooling lets the model adapt within a few frames when several cars are visible, faster than per-agent estimation and faster than online identification. The advantage should vanish when only one car is visible, a falsifiable signature of the pooling mechanism.

A second experiment directly tests the Exarchos et al. finding. We train otherwise identical predictors with geometry-only randomization, grip-only randomization and both, then evaluate on held-out grip and held-out geometry. For cars at speed we expect grip randomization to dominate, contrary to the quadruped result. If geometry randomization wins, that would itself be a notable finding about forecasting.

Finally, the Assetto Corsa human laps test whether grip inferred from history predicts where expert drivers at the limit actually go. The nuScenes check confirms the physics layer costs little on ordinary traffic.