Least-Favorable Mode Weights for Robust Multi-Modal Trajectory Forecasting

Forecasting where a car or pedestrian will go is usually framed as predicting a small set of candidate futures, such as "turn left," "go straight," or "brake," each with a probability. A motion planner then decides how much to fear each one. These probabilities are brittle. Moving an agent's observed past by less than a meter, in ways that still obey realistic speed and jerk limits, can make a forecaster swing confidently toward a lane change that will not happen. We propose to replace the single probability vector with an interval of plausible probabilities for each future, sized by how far small input disturbances could push it. From that interval we report the one distribution that classical robust estimation identifies as optimal against the worst case. For a discrete set of futures, this distribution has a closed form: the nominal weights, clipped from above and below.

The idea borrows a specific result rather than the general slogan "be robust." In minimax filtering, an estimator is designed to minimize its worst-case error over a set of plausible noise models rather than its average error under one assumed model. Minimax Linear Smoothing for Capacities proves that when spectral uncertainty is described by a 2-alternating capacity, the minimax filter is the nominal Wiener filter clipped between two constants. A 2-alternating capacity is a set-valued generalization of a probability that covers contamination and total-variation neighborhoods. Probability intervals on a finite set of outcomes are themselves a capacity of this kind. Carried over to the predicted futures, the result implies that a forecaster facing disturbed inputs should clip its mode probabilities in an input-dependent way. Rare but reachable futures get a floor, and over-confident ones get a ceiling. We build this, make it cheap enough to run in one forward pass, and test whether it beats adversarial training and simple probability floors.

Core Claim

Our hypothesis is that much of the damage reported under realistic attacks on multi-modal forecasters comes from probability mass moving between modes, rather than from distortion of each mode's path. If so, a forecaster that outputs the least-favorable mode distribution over an input-dependent credal set (a set of candidate probability vectors) should keep most of its nominal accuracy. It should also sharply reduce the worst-case probability it assigns to the true future, which neither geometric smoothing nor certification of output tubes addresses.

Concretely, let $\pi(X)$ be the forecaster's probabilities over $K$ discrete modes given observed history and context $X$. Let $\Delta$ be the set of kinematically feasible history perturbations of up to 1 m used by On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles. The credal set $C(X)$ consists of probability vectors $p$ with $l_k(X) \le p_k \le u_k(X)$, where $l_k$ and $u_k$ are the smallest and largest values of $\pi_k(X+\delta)$ over $\delta \in \Delta$. Under log-loss, the minimax predictor $\min_q \max_{p \in C(X)} \mathbb{E}_p[-\log q]$ is the maximum-entropy member of $C(X)$. It has the water-filling form

$$p^*_k = \max\{\, l_k(X),\ \min\{\, u_k(X),\ c \,\}\}, \qquad \textstyle\sum_k p^*_k = 1,$$

where the single level $c$ is found by a one-dimensional search. This is the discrete analogue of Poor's clipped transfer function. When the intervals are narrow, the output equals the nominal forecast. When an input region is fragile, probability flows toward the futures the adversary could have promoted.

Research Program

The backbone is Trajectron++. We use it because it already separates a discrete latent "intent" variable, a conditional variational autoencoder whose categorical latent plays the role of the mode, from a kinematic decoder that turns each mode into a feasible path. It is also the model shown to be most exposed to attack. We keep its per-mode path decoder and add an interval head. This head predicts $l_k(X)$ and $u_k(X)$ from the same encoder features.

The head is trained by amortized inner maximization. For each training batch, a projected-gradient attacker searches $\Delta$ for perturbations that minimize and maximize each mode's probability. The attacker uses the velocity, acceleration, and jerk constraints and the lane-change and braking objectives of the adversarial-robustness study. The resulting extremes supervise the interval head through a pinball loss that penalizes under-coverage more than over-width. The outer objective is the log-likelihood of the observed future under $p^*$, plus the backbone's usual reconstruction term for paths. The water-filling map is piecewise linear, so gradients reach both the nominal weights and the interval bounds.

At test time no attacker runs. The cost is one forward pass and an $O(K \log K)$ clip. This matters relative to the many forward passes per agent that randomized smoothing needs in Certified Human Trajectory Prediction.

We will not claim certification. Instead we calibrate. On held-out scenes we run a stronger attacker, with more steps, restarts, and the black-box particle-swarm search from the adversarial-robustness study. We measure how often it pushes $\pi_k$ outside the predicted interval, then inflate interval widths by a split-conformal factor until coverage reaches a target such as 95%. The guarantee is therefore empirical and specific to the attack family. We state that limitation plainly.

A second disturbance family tests whether the construction generalizes beyond history perturbations. This family is withholding or corrupting map context, the failure mode documented in Beyond ADE and FDE. There, AgentFormer's final displacement error rises from 3.89 m to 4.21 m without the map, and to 5.41 m on curved roads. We add map-dropout extremes to the interval targets and test on the curved-road, no-map slice.

Baselines

The naive reference is the unmodified Trajectron++ trained by its standard objective. It is evaluated both on clean inputs and under the attack protocol. The adversarial-robustness study reports that attacks raise displacement error by more than 150% on average across the models it tested, including Trajectron++. Our reproduction must recover a degradation of that order before any comparison is meaningful.

The empirical defenses from that same study are data augmentation and linear trajectory smoothing. They are reported to cut error under attack by about 28% while raising error on normal aggressive driving by 6% to 28%. Beating them means a larger reduction in worst-case error on the true mode, with a nominal cost below 6%.

The strongest competitor is standard adversarial training. This is the same backbone, trained on the inner attacker's worst-case perturbations of the full output, with no interval head. It is the method a reviewer would expect to subsume ours. It gets an identical attacker, attack budget, number of training epochs, and hyperparameter search budget, and it is evaluated on the same held-out attacker. No reported result was found on Lacuna for adversarially trained Trajectron++ under the kinematically constrained attack, so we must produce this number ourselves. Beating it means a lower worst-case negative log-likelihood of the true mode under attack at equal or better clean minimum average and final displacement error.

The certified alternative is randomized median smoothing with a diffusion denoiser, from Certified Human Trajectory Prediction. That paper reports the denoiser shrinking the final bound half-diameter from 1.20 m to 0.78 m at a fixed error rate. We run it on the same backbone with its published sample counts. We compare worst-case mode probability, displacement error, and wall-clock time per agent. We do not expect to beat its formal guarantee. The question is whether we match its practical robustness at a small fraction of its cost.

Two ablations isolate the key ingredient. The first is a constant floor and ceiling on mode probabilities, tuned so that clean log-likelihood matches ours. It tests whether input-dependent intervals matter at all. The second applies our water-filling clip to adversarially trained weights while removing the interval head. It separates the benefit of the clipped output form from the benefit of learned interval widths.

Why not just…?

Why not just do adversarial training? This is the real threat. Adversarial training already solves a min-max problem, and if it removes most of the mode-flipping, our construction adds little. The difference is in what is optimized. Adversarial training lowers the loss at the single worst perturbation it finds. It does not stop the reported weights from moving within $\Delta$, so a planner still sees confident swings under perturbations the attacker missed. Our output is constructed to sit at the least-favorable point of the whole interval. The head-to-head comparison under the held-out stronger attacker, with matched budgets, settles this. If adversarial training matches us on worst-case true-mode likelihood, we will report that the clipped form is unnecessary.

Why not just put a probability floor on every mode, or use temperature scaling? A constant floor is the simplest version of our clip. Its weakness is that it spends probability on implausible modes in scenes that are not fragile, and spends too little in scenes that are. That should show up as worse clean likelihood or worse attacked likelihood than ours at matched calibration. The constant-threshold ablation tests this directly. If it matches our method, the contribution reduces to "add a floor," and we will say so.

Why not just smooth the inputs or certify the outputs? Linear smoothing does not stop kinematically legal semantic attacks such as spoofed lane changes, and it degrades nominal error by up to 28%, according to the adversarial-robustness study. Randomized smoothing certifies spatial tubes around the outputs, but it needs many forward passes per agent. It also bounds where each path can move, not how probability is reassigned among paths, which is the quantity a planner uses to weigh which future to fear. The comparison with randomized smoothing reports both quantities and the timing.

Proposed Experiments

The first check is diagnostic and can kill the project. On nuScenes and Apolloscape, under the kinematically constrained attack, we split the attacked displacement error of Trajectron++ into two parts. One part comes from re-ranking modes, measured by scoring attacked weights with clean paths. The other comes from per-mode geometric distortion, measured by scoring clean weights with attacked paths. If geometry accounts for most of the degradation, clipping the mode weights cannot help much. In that case we redirect the work toward per-mode set-membership bounds.

If re-ranking dominates, the main experiment trains all baselines on nuScenes and on ETH/UCY. We report clean minimum average and final displacement error over the top predicted modes. We also report worst-case negative log-likelihood of the true mode and the rate at which a rare true mode falls below 5% probability. Every metric is reported clean, under the white-box attack, under the black-box attack, and on the curved-road no-map slice.

The interval calibration is checked by measuring empirical coverage against attackers not used in training.

A planning-relevance check weights forecast errors with the differentiable sensitivity measure of Rethinking Trajectory Forecasting Evaluation. That work shows identical displacement errors differing in planning impact, with sensitivity scores of 0.90 versus 0.21 in a head-on scenario. This check asks whether our gains concentrate on errors that matter to the planner.

Success means three things relative to adversarial training at equal budget. Our method must have lower worst-case true-mode likelihood under the held-out attacker, and must lose less than 6% in clean displacement error. It must also beat the constant-floor ablation on both axes.