Learning Only From Surprises: Single-Pass Gating for Online Adaptation of Multi-Modal Trajectory Forecasters

Self-driving cars and robots predict where nearby people and vehicles will go, and they do it with models trained offline on data from some cities, some weather, and some driving cultures. When they are deployed somewhere new, accuracy drops. One lucky property of this problem is that the correct answer arrives for free a few seconds later, because the car simply watches where each agent actually went. A forecaster could therefore keep learning on the road. The hard part is deciding which of these delayed answers to learn from. Visual object tracking learned this lesson long ago: trackers that update on every frame absorb occluders and noise and often end up worse than trackers that never update. Trajectory forecasting adds a twist that tracking does not have. The future is genuinely branching. A car at an intersection may turn left or go straight, and a large error is often not a sign that the model is broken. It may only mean the agent took the less likely branch the model had already allowed for.

We propose a gate that decides when a deployed forecaster should take a gradient step. The gate fires only when the observed error is surprising given the model's own predicted uncertainty, and only when the input looks like a sustained shift in the world rather than a one-off sensor glitch. Every quantity the gate uses comes from a single forward pass, so the whole scheme fits a real-time budget, unlike gates built on many stochastic forward passes or model ensembles. The claim is that, at a fixed and small update budget, surprise-gated adaptation recovers more accuracy under real distribution shift than updating always, updating on large raw errors, or updating at random. We further claim it does this without collapsing the forecaster's multiple predicted futures into one.

Core Claim

Raw error is the wrong trigger for online learning in multi-modal forecasting. "Multi-modal" here means the model outputs several distinct candidate futures with probabilities, rather than one averaged path. These models are typically trained with a winner-take-all loss, which updates only the candidate closest to what happened. Under that loss, gradient steps on high-error samples mostly come from rare branches and from aleatoric randomness, meaning randomness that is inherent in the data and cannot be learned away. Updating on such samples drags probability mass toward rare branches or erodes the diversity of predicted futures. The useful signal is the gap between realized error and expected error. When that gap is large across many inputs that the training data did not cover, the model is epistemically wrong: it lacks knowledge it could acquire. This gap can be computed in one pass from cheap auxiliary heads that already exist in the literature.

Concretely, we take the post-hoc modules from Joint Out-of-Distribution Detection and Uncertainty Estimation for Trajectory Prediction, which supplies the method we reuse. That paper attaches a Gaussian mixture density over frozen encoder features and a small error-regression network to the HiVT forecaster, reporting a 2.8% AUROC gain for out-of-distribution detection and 10.1% R-AUC on the Shifts benchmark at only 0.14 ms added latency. The authors leave their system passive: it flags shift but never acts on it.

We turn those flags into an update rule. For a scene seen at time $t$ whose outcome is revealed at $t+H$, where $H$ is the forecast horizon, let $e_t$ be the realized minimum displacement error over the predicted futures. Let $\hat e_t$ be the error the regression head predicted at time $t$, and let $d_t$ be the log-density of the scene's encoder features under the training-set mixture. The gate is

$$ g_t = \mathbb{1}\left[\frac{e_t}{\hat e_t} > \tau_s\right] \cdot \mathbb{1}\left[q_{\text{train}}(d_t) < \tau_{\text{shift}}\right] \cdot \mathbb{1}\left[q_{\text{stream}}(d_t) > \tau_{\text{glitch}}\right]. $$

In this rule, $q_{\text{train}}$ is the quantile of $d_t$ among training scenes, and $q_{\text{stream}}$ is its quantile among the last few thousand deployment scenes. The thresholds $\tau$ are set as quantiles rather than raw scores.

The first factor asks whether the error is a surprise. The second asks whether the world has shifted relative to training. The third asks whether this scene is an outlier even relative to the recent stream, which would suggest a perception glitch rather than a new regime. This is a direct transplant of the "high gain" band from Continuous convolutional object tracking, which updated a tracker only when its confidence lay between 0.2 and 0.4. That work skipped both hopeless frames and already-mastered ones, and it raised precision on the Online Object Tracking Benchmark from 0.653 with no updates to 0.840. Two things are different here. The band is defined against two moving reference distributions instead of fixed confidence values, which addresses the threshold brittleness that work leaves open. And the "already mastered" side of the band is defined by surprise rather than by confidence.

Research Program

The backbone is HiVT-64 from HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction. It is a 662K-parameter forecaster running at about 20 ms per scene that outputs a mixture of Laplace distributions over futures. We chose it so that our results stack directly on the HiVT* auxiliary heads.

The encoder and both auxiliary heads stay frozen. This choice is deliberate. If the features the gate reads were themselves adapted, the gate could be fooled by the adaptation it controls: a drifting encoder can make every input look familiar. We therefore adapt only low-rank residual adapters inserted in HiVT's global interaction module and its output head, together with the mixture weights. The gate, in contrast, reads only the frozen local-encoder features. When the adapted forecaster improves, $e_t$ falls toward $\hat e_t$ and the gate closes on its own. This gives a homeostatic stopping rule, meaning one that stops updates once the model is doing as well as expected, without any explicit schedule.

Gated samples go into a small first-in-first-out buffer, again following the cache design of the tracking work above. Every $k$ steps, one gradient step is taken on a mini-batch from the buffer, using the standard winner-take-all and negative log-likelihood loss. This backward step runs off the critical inference path, so forecast latency equals HiVT's single forward pass plus the 0.14 ms of the heads. Because labels are true future positions rather than pseudo-labels, no teacher network is needed. That removes the Fisher-information computation and the augmentation generator that Human Motion Forecasting in Dynamic Domain Shifts: A Homeostatic Continual Test-Time Adaptation Framework runs at test time.

The analysis component tests the mechanism, not just the headline number. For every gate, we log which mixture components receive gradient and how the entropy of the predicted mixture weights evolves. We also log how often updates land on scenes later judged to be in-distribution. This is how we will check the claim that error-triggered updates erode mode diversity.

On the role of the four directions: out-of-distribution generalization defines the regime, and multi-modal forecasting is the object being adapted. The online appearance-adaptation literature supplies the stability-versus-plasticity lens and the gating design. Stochastic inference overhead is a hard constraint, satisfied because every gate quantity is single-pass. We do not adapt pixel-level appearance in the main study. As a secondary experiment, we will apply the same gate to the convolutional map encoder of Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data on nuScenes. That model's visual encoder is static, and it decodes controls through a unicycle model, which keeps adapted predictions kinematically feasible.

Baselines

The naive reference is the frozen HiVT forecaster with the HiVT* heads, which is exactly the passive system of Joint Out-of-Distribution Detection and Uncertainty Estimation for Trajectory Prediction. It is the strongest existing method on Shifts for single-pass uncertainty. Its reported gains are relative: +2.8% AUROC, +10.1% R-AUC, and 0.14 ms overhead. We found no absolute HiVT minADE on Shifts on Lacuna, so we will re-measure it ourselves under the identical streaming protocol.

The strongest adaptive competitor is ungated continual updating of the same adapters on every revealed outcome. This isolates the gate from the adapter parameterization. We also run EANet from EANet: Expert Attention Network for Online Trajectory Prediction, which shifts attention between fast-learning shallow layers and slow deep layers and reports 27.19% (SNU) and 31.35% (SDD) ADE/FDE reductions after 1,000 instances at 30 frames per second. EANet is the reference online trajectory learner without an uncertainty safeguard, and we compare against it on SDD under its own protocol.

The ablations that isolate our key ingredient run at a matched update budget, meaning the same number of gradient steps per stream. The four gates compared are: random selection, raw-error gating (the top fraction of $e_t$), density-only gating (the second factor alone), and the full surprise gate. A stochastic upper reference replaces $\hat e_t$ and $d_t$ with variance from ten Monte Carlo dropout passes, following Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, and a 5-model ensemble serves as a further upper reference. Both are reported with their measured latency.

Fairness is enforced in four ways. All methods share the same streams, the same adapter capacity, and the same buffer size. Gate thresholds are tuned on a held-out shifted validation stream from a different city than the test streams, with an equal tuning budget for each method. Wall-clock time is measured on one GPU.

Beating the field means two things. Our gate must give lower cumulative stream minADE than raw-error and random gating at the same budget. It must also match the dropout and ensemble gates to within noise at a small fraction of their latency.

Why not just…?

Why not just update on every revealed outcome, since the labels are true futures rather than pseudo-labels? This is the most serious threat. The tracking finding that full updates can be worse than none came from noisy self-generated labels, which do not exist here. Our answer is empirical and two-sided. First, under a compute budget that permits updates on only a fraction of steps, which choices matter is the question. Second, revealed futures are only as clean as upstream perception, and the stream mixes rare branching outcomes in with genuine shift. If always-update matches our gate at equal steps, and also preserves mixture entropy and in-distribution accuracy after the stream returns to source conditions, then the gate's value reduces to saving compute, and we will report it that way.

Why not just gate on large raw error, which needs no auxiliary heads? Raw error cannot tell a model that is wrong apart from a future that was hard to predict. Under winner-take-all training, updates on rare-branch outcomes should collapse or skew the mixture. The decisive check is an in-distribution control stream. There, a correct gate should almost never fire. Raw-error gating will fire on the most aleatoric scenes, and we will measure whether this degrades miss rate and mixture entropy.

Why not just use an ensemble or Monte Carlo dropout to decide when to update? These are plausibly the best possible gates and are included as upper references. The source paper reports that a 5-model ensemble adds 400% compute, and dropout scales cost with the number of passes, both of which exceed the per-scene budget a 20 ms forecaster leaves. Our result is meaningful only if the single-pass gate captures most of their benefit.

Why not just train offline on more diverse data? Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks shows that models trained on large industrial data still degrade when moved to new cities and weather. Generative Causal Representation Learning for Out-of-Distribution Motion Forecasting learns invariant versus environment-specific factors offline and holds ADE at 0.97 where a baseline reaches 2.68 under heavy noise. That approach is complementary rather than competing, and we will apply our gate on top of it on ETH-UCY to test whether offline invariance leaves residual shift for online learning to fix.

Proposed Experiments

The primary benchmark is the Shifts vehicle-motion task, trained on its in-distribution split and streamed in timestamp order through its shifted partitions (other cities and precipitation). Outcomes are revealed after the forecast horizon. We report cumulative online minADE, minFDE and miss rate, CNLL, R-AUC of the uncertainty heads after adaptation, update fraction, and per-scene latency. We also report forgetting, measured as accuracy on a source-domain stream replayed after adaptation; HoCoTTA's 119.2 mm versus 127.0 mm full-update gap on Human3.6M→GRAB suggests forgetting is real in motion forecasting.

Secondary streams are ETH-UCY leave-one-scene-out and SDD, the latter for a direct comparison with EANet's reported recovery figures. A third secondary experiment applies the Trajectron++ map-encoder variant on nuScenes, with photometric corruption injected into map and camera inputs as a proxy for appearance shift.

Three checks decide the claim. The first is a matched-budget curve of stream minADE against update fraction for the four gates. The second is the in-distribution control stream, which measures spurious firing and mixture-entropy loss for each gate. The third is a glitch-injection test: a few percent of tracks receive spoofed jitter, and we count how many gradient steps land on the corrupted samples under each gate. If the surprise gate wins the first check only by matching raw-error gating, and loses nothing on the second, the multi-modal argument is wrong and the note's contribution narrows to cheap, safe gating against glitches.