Self-driving cars must forecast where nearby cars and pedestrians will go, often for agents seen for only a fraction of a second: a car that just came out from behind a truck, or a pedestrian the tracker briefly lost. Recent methods handle these short histories by teaching the short-history model to imitate what a model with the full two-second history would compute internally. This note argues that this imitation target is wrong. Many different long pasts are consistent with the same short glimpse. Forcing the short-history representation toward the long-history representation of each training example pulls it toward an average long-history representation. That is the same "averaging trap" that sends deterministic forecasters into the median between a left turn and a right turn, now happening one layer earlier.
The proposed fix is simple. Distill at the level of predicted distributions, not hidden features, using the direction of divergence whose minimizer is provably the right answer: the mixture of all the long-history forecasts compatible with the short glimpse. If the argument is right, shorter histories should yield predictably wider, better-calibrated forecasts. Existing methods should instead look confident and be wrong more often in ways that the standard "best of six guesses" metric cannot detect.
Background and the gap
Forecasting is multi-modal: the model outputs several weighted candidate futures, typically a mixture of Gaussian or Laplace distributions. A single regressed path averages mutually exclusive choices into impossible ones, as Multiple Futures Prediction documents. Our base model is HiVT, a 662K-parameter map-aware transformer that outputs a Laplace mixture on Argoverse. It supplies the backbone; nothing in its design is in question.
Three recent works tackle observation length shift, meaning models trained on full histories collapse when given 2–5 frames.
CLLS/RNLS uses a contrastive loss that pulls together embeddings of two random temporal crops of the same track. It finds that the last two frames carry most of the short-range signal, but it uses no HD maps.
LaKD uses one shared backbone that distills features between short and long versions of the same track. The direction flips to whichever version has lower error. Gradient-based soft-masking of weights limits "knowledge collision," where short-history training damages long-history accuracy.
PRF uses a cascade of units that align features to slightly longer horizons and explicitly reconstruct missing past steps. It reports top results on the Argoverse 2 leaderboard, but its authors concede failures below about 10 steps in U-turns and merges.
These three methods differ in mechanism. All three motivate the proposal, and all three share the feature this proposal targets: a per-example alignment between a short-history representation and that example's long-history representation (LaKD, PRF) or an invariance between crops (CLLS). All three also report only best-of-K displacement and miss rate. Those metrics reward coverage and cannot detect overconfident mode probabilities.
Core Claim
Let $s$ be the short observed history, $\ell$ the full history it was cut from, and $Y$ the future. The correct short-history forecast is the marginal $$p(Y \mid s) = \mathbb{E}_{\ell \sim p(\ell \mid s)}\big[\,p(Y \mid \ell)\,\big],$$ a mixture over all pasts consistent with $s$. A frozen full-history teacher $p_T(Y\mid \ell)$ and a student $p_S(Y\mid s)$ can be trained with the forward-KL objective $$\min_S \; \mathbb{E}_{(s,\ell)}\; \mathrm{KL}\big(p_T(\cdot\mid \ell)\,\|\,p_S(\cdot\mid s)\big).$$ For each $s$ this objective is minimized exactly by the teacher's mixture above. This is the standard property that forward KL averaged over conditioning variables yields the mixture. The student becomes appropriately wide where the short history is genuinely ambiguous.
By contrast, a squared-error loss between student and teacher features is minimized by the mean teacher feature. That mean feature need not decode to any of the compatible futures, which is the latent-space averaging trap.
In practice the forward KL between mixtures is computed by sampling futures from the teacher's mixture and maximizing their student log-likelihood. This is cheap and needs no closed form.
The testable consequence is specific. Feature-aligned methods should show predicted entropy that stays roughly flat as the history shrinks, and their likelihood and probability-weighted errors should be worse. Marginal distillation should show entropy rising as history shrinks, with calibrated mode probabilities. Best-of-K error may be similar or better.
Research Program
Building the models. We train a full-history HiVT teacher, freeze it, and initialize a student from it. The student is trained on rolling-start and random-truncation crops, with crops of 2 to 20 frames on Argoverse 1. The loss combines ground-truth mixture likelihood with the sample-based forward-KL term. Because the teacher is frozen and the full-history path can be served by the teacher itself, knowledge collision cannot occur by construction. This removes the need for LaKD's soft-masking.
The student keeps HiVT's map and neighbor attention unchanged. Missing frames are handled by its attention masks rather than padding. This also addresses the map gap CLLS leaves open.
Diagnosing the latent-space averaging trap. We will show the trap directly on held-out Argoverse scenes, using pairs that are near-identical over their last few frames but differ in earlier history. One example is a car that was in a turn lane two seconds ago versus one that was not. For such pairs we compare three things: the teacher forecasts on each long history, the student forecast on the shared short history, and the forecast decoded from the averaged teacher feature. The trap exists if the averaged feature decodes to paths that no long-history forecast supports, such as off-lane or between maneuvers.
Calibration across lengths. At every observation length we report:
- minADE, minFDE and miss rate at K=6;
- mixture negative log-likelihood;
- probability-weighted final error, i.e. the Brier-style minFDE used on Argoverse 2;
- calibration error of the top-mode probability;
- predicted entropy as a function of length.
Baselines
Naive reference points. The first is a HiVT trained on full histories and evaluated on truncated ones, with missing frames masked. This is the "do nothing" option. The second is a constant-velocity extrapolation from the last two frames, which tests how much of the short-history regime is just kinematics. Beating these is necessary but not interesting.
Random masking with likelihood loss. This trains HiVT on randomly truncated histories with its usual mixture likelihood. It is the strongest simple competitor, because ground-truth likelihood on short inputs is in principle also an unbiased route to the marginal. LaKD reports that it suffers knowledge collision. It gets the same crops, epochs and tuning budget as our student. Our method must beat it on likelihood and calibration at short lengths, not just match it.
LaKD on HiVT. This is the strongest existing method with a reported number on our exact backbone. LaKD reports HiVT minADE (K=6) on Argoverse 1 of 0.7255 for the original and 0.6807 with LaKD. We reproduce it with matched compute and the same truncation protocol. Beating it means at least matching its minADE per length while improving NLL, Brier-style minFDE and top-mode calibration at 2–5 frames.
PRF. PRF leads the Argoverse 2 leaderboard when paired with the DeMo backbone. However, no reported scalar result was found on Lacuna for PRF on Argoverse 1 or 2. We therefore rerun it on HiVT under our protocol and compare on the same per-length metrics.
Key ablation. The same frozen teacher, student initialization and data are used under three losses: feature squared-error alignment, reverse-KL distribution distillation, and forward-KL distribution distillation. This isolates the claim. If feature alignment matches forward KL on calibration, the core claim is false.
Upper reference. Separate specialist HiVTs, one per observation length, bound what a single student can achieve.
Why not just…?
Why not just train with random masking and the ordinary likelihood loss, since it already targets the marginal? This is the real threat, because in expectation it optimizes the same quantity. The difference is variance. Each training example supplies one sampled future, whereas the teacher supplies a full distribution, which is the usual reason distillation beats raw labels. LaKD's random-masking results suggest the shared-weight version also degrades long histories. The baseline comparison above settles this. If random masking matches our student at every length, the contribution narrows to a diagnosis of LaKD- and PRF-style alignment, and we will say so.
Why not just recalibrate mode probabilities after training, with per-length temperature scaling? Temperature scaling reweights existing modes. It cannot move mode locations or widen spatial spread when the short-history model has placed its six paths around an averaged intent. We will apply per-length temperature scaling to every baseline. Our claim survives only if gains in likelihood and probability-weighted error persist after it.
Why not just reconstruct the missing past, as PRF does? PRF's mode queries do generate several past hypotheses, and if those hypotheses are carried into the future forecast, PRF marginalizes implicitly. Its feature-alignment module, however, still regresses toward the specific long history. Its reported failures at ultra-short lengths in U-turns and merges are the cases where many pasts are compatible, which is what our argument predicts. The per-length calibration comparison, restricted to those scene types, tests this directly.
Why not just keep one model per length? It is an honest upper reference. It multiplies storage and must switch models as tracks grow. If specialists beat our single student by a wide margin, the single-model framing is not worth it.
Proposed Experiments
The primary benchmark is Argoverse 1, with histories truncated from 20 frames down to 2 and a 3-second horizon on the standard splits. Argoverse 2 and nuScenes serve for generalization, matching the settings LaKD used.
The decisive check has two parts. First, at 2–5 frames, forward-KL distillation should improve NLL and top-mode calibration over feature alignment, LaKD and PRF, with predicted entropy rising monotonically as history shrinks. Second, the paired-scene analysis should show feature-averaged decodes landing on paths unsupported by any compatible long history.
A secondary check is whether best-of-K error stays within noise of LaKD's 0.6807. If it does, calibration is gained without giving up the leaderboard metric. A clear loss would mean the wider distributions are buying honesty at the cost of coverage, and we will report that trade-off.
References
- Multiple Futures Prediction
- HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction
- Adapting to Observation Length of Trajectory Prediction via Contrastive Learning
- LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations
- Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction