Self-driving systems predict where nearby cars will go over the next few seconds. Real driving logs with good labels are costly to collect, while simulated scenes are nearly free. Simulated traffic, however, is driven by simple scripted planners. Recent work shows that pre-training a forecaster on synthetic scenes and then fine-tuning it on real data helps. Fine-tuning has two problems, though. It can wipe out what the model learned in simulation, including rare road shapes it will meet again on real roads. It also keeps whatever simulator habits the real data happens not to correct. This note proposes a different handoff. We train a "teacher" forecaster purely in simulation. A "student" then learns from real data while also copying the teacher. The student copies only the parts of the teacher's knowledge that survive the move to reality, and it copies them in stages.
The bet rests on two observations from separate literatures. First, a teacher's earlier training snapshots hold simpler, more general skills than its final version (Progressive distillation induces an implicit curriculum). Second, forecasters that output the whole future path in one pass, called non-autoregressive or parallel decoders, produce outputs that can be addressed separately: one per future time step and one per candidate destination. We hypothesize that simulator knowledge about road geometry and short-horizon motion transfers to real traffic, while knowledge about long-horizon intent and how likely each maneuver is mostly encodes the scripted planner. If so, a schedule that moves through teacher snapshots and chooses, output by output, what to copy should beat both fine-tuning and ordinary one-shot distillation. The advantage should be largest when real data is scarce and on rare road layouts.
Core Claim
Simulator knowledge in trajectory forecasting can be separated along two axes. The first is training time: which teacher checkpoint it comes from. The second is output structure: near-horizon waypoints and destination geometry on one side, far-horizon endpoints and how the model ranks maneuvers on the other. Parallel decoders make the second axis directly controllable. In an autoregressive decoder, which feeds each predicted step back in to predict the next, copying the teacher's step 30 also means copying the path that led there. A parallel decoder has a separate output for each step, so the loss on each can be weighted independently. The resulting student should keep simulator competence on rare geometry while reducing what we call "planner-likeness": the tendency to predict the scripted planner's constant-speed, lane-centered motion in real scenes.
Research Program
The synthetic data follows the recipe of Pre-training on Synthetic Driving Data for Trajectory Prediction. That work bends real HD-map lanes into sharp curves, generates paths with a rule-based A* planner (about 370k scenes), pre-trains with masked reconstruction, and fine-tunes DenseTNT on Argoverse. It supplies our data pipeline and our strongest baseline. It does not use distillation, so simulator knowledge is not protected once fine-tuning begins.
The teacher and student backbone is HiVT (HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction). HiVT is a parallel multi-agent forecaster that outputs several candidate futures at once. Its 64-dimension version has 662K parameters and runs in about 20 ms. We also run DenseTNT (DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets) for direct comparison with Li et al. DenseTNT first scores a dense grid of candidate destinations and then generates a path to each chosen one. This gives a natural split between destination geometry (where the drivable endpoints are) and the scores it assigns to those destinations.
We train the teacher only on synthetic scenes and save checkpoints throughout training. A transferability probe then evaluates every checkpoint on a small held-out slice of real data. The probe scores each output component separately: waypoint error at each future step, accuracy of the destination heatmap support, and calibration of the mode probabilities. From these scores we build a schedule. Stage $s$ distills from checkpoint $c(s)$. Early stages use early checkpoints and copy all components. Later stages use later checkpoints but copy only the components the probe marked as transferable. The student objective is
$$\mathcal{L} = \mathcal{L}_{\text{real}} + \sum_{h} \lambda_h(s)\, D\big(f^{S}_h(x),\, f^{T_{c(s)}}_h(x)\big),$$
where $\mathcal{L}_{\text{real}}$ is the usual supervised loss on real Argoverse trajectories, $h$ indexes output components (per-step waypoints, destination heatmap, mode scores), $f^S_h$ and $f^{T_{c(s)}}_h$ are the student's and stage-$s$ teacher's outputs for component $h$, $D$ is a Laplace negative log-likelihood or KL divergence, and $\lambda_h(s)$ is the stage-dependent weight set by the probe.
Distillation inputs come mostly from synthetic scenes, especially the warped high-curvature ones that real data rarely contains. A smaller share comes from real scenes run through the teacher. The real-data loss stays on for the whole run. This answers the objection raised by Privileged Knowledge Distillation for Sim-to-Real Policy Generalization, which argues that two-stage teacher-student imitation produces overly conservative locomotion policies and favors single-stage joint training. Our student never imitates the teacher alone. We also include a single-stage joint co-training baseline in the spirit of that paper.
Three lines of work set the context. TWIST distills a simulator teacher with privileged state into a vision student in staged fashion for real robot pushing and lifting. It shows that staged sim-to-real distillation can work (80% vs 30% real Block Push success over domain-randomized Dreamer). However, its dynamics are autoregressive, it copies everything the teacher knows, and it does not address forecasting other agents. Enhancing Mapless Trajectory Prediction through Knowledge Distillation and LaKD distill inside parallel forecasters, from a map-using teacher and across history lengths respectively. Both work entirely on real data, in one stage, without regard to domain. Teacher-assistant chains (Improved Knowledge Distillation via Teacher Assistant) and annealed matching (Continuation KD) target the capacity gap between teacher and student. We use their staging idea but aim it at a domain gap instead: teacher and student share an architecture, so what the stages bridge is simulator behavior, not model size.
Baselines
The naive reference is HiVT, and separately DenseTNT, trained from scratch on real Argoverse 1 only. LaKD reports HiVT at minADE (K=6) of 0.7255 on Argoverse 1, which gives us a sanity anchor for our from-scratch reproduction.
The strongest existing method is synthetic pre-training followed by fine-tuning, as in Pre-training on Synthetic Driving Data for Trajectory Prediction. Over DenseTNT from scratch on Argoverse validation, that paper reports relative gains of 8.30% in minADE₆, 3.84% in minFDE₆ and 5.04% in MR₆. No absolute validation number for this baseline was found on Lacuna, so we reproduce it ourselves. For reference, the DenseTNT paper reports leaderboard miss rate falling from about 8.2% to 7.0%. This baseline is the main threat because it uses the same synthetic data and the same architecture, and adds no extra machinery. To keep the comparison fair, it gets the same synthetic data and the same total GPU hours, which covers teacher training and probing on our side. It also gets a tuning budget equal to ours, spent on learning-rate sweeps, layer freezing, and a regularization term that keeps weights close to the pre-trained ones.
The third baseline is one-shot distillation: the student copies all outputs of the final synthetic teacher alongside the real loss, which applies FOKD-style output distillation across domains. The fourth is single-stage joint co-training on mixed synthetic and real batches with a tuned mixing ratio, following the logic of HIB.
The ablations that isolate the key ingredients are checkpoint staging with all components copied, component selection from the final checkpoint only, and the best single early checkpoint with component selection. Beating the fine-tuning baseline means a minFDE₆ and MR₆ improvement larger than three-seed standard deviation at 10% and 25% real-data fractions. It also means a clear gain on the high-curvature and sparse-traffic slices, with no loss at 100% real data.
Why not just…?
Why not just fine-tune the pre-trained model more carefully? This is a real threat. Fine-tuning gave consistent gains in Li et al., and a good weight-proximity regularizer partly protects simulator knowledge. It cannot choose which knowledge to protect, though. Holding weights near their pre-trained values keeps planner habits and road geometry together. Our planner-likeness diagnostic and the curvature-slice results will show whether that difference matters. If tuned fine-tuning matches us at every data fraction, the claim fails and we will report that.
Why not just distill from the final simulator teacher in one shot? The final checkpoint is where scripted-planner shortcuts should be most firmly memorized. Both the progressive-distillation results and the warning in the CARLA benchmark (CARLA: An Open Urban Driving Simulator) that learned policies overfit to non-player traffic behavior point that way. The one-shot baseline and the "final checkpoint plus selection" ablation test this directly.
Why not just mix synthetic and real data in one training run? Mixing gives the model no means of separating planner behavior from geometry: every synthetic label is treated as true. It is the cheapest alternative and HIB favors single-stage training, so we tune its mixing ratio with the same budget we give our own method.
Why not just pick one good early checkpoint? This might be enough, in which case the multi-stage part is unnecessary. The "best single early checkpoint" ablation settles it. Gaining less than seed noise from the full schedule over that ablation would cut the proposal down to checkpoint selection plus component selection.
Proposed Experiments
The main experiment trains on 5%, 10%, 25% and 100% of the Argoverse 1 training set (Argoverse: 3D Tracking and Forecasting With Rich Maps). We report minADE₆, minFDE₆ and MR₆ on validation, both overall and on slices for high curvature, sparse traffic (few neighboring agents, where the FOKD student was reported to collapse), and short observed histories. Two further measurements check the hypothesis itself. The first is planner-likeness: on real scenes, how far predicted speed and lateral profiles deviate from an A* centerline reference. The second is jerk, as a check on kinematic feasibility.
A mechanism check comes from the per-component, per-checkpoint transfer curves. Near-horizon waypoint error and heatmap support should transfer from late checkpoints. Mode-score calibration should transfer worse as the teacher trains longer. If that ordering does not appear, the premise behind component selection is false.
An architecture check replaces HiVT's parallel head with an autoregressive recurrent decoder of the same size. We then try the same component selection there, where it can only weight steps through the rollout. A gap that shrinks or disappears would support the claim that parallel decoding is what makes selective distillation possible.
A transfer check evaluates the Argoverse-trained students zero-shot and with light adaptation on Argoverse 2. This tests whether keeping simulator geometry helps in cities never seen in training. The decisive result is the 10–25% regime. The full method must beat tuned fine-tuning there, both overall and on curvature slices, by more than seed variance, while showing lower planner-likeness. Otherwise the idea is not worth its extra stages.
References
- Progressive distillation induces an implicit curriculum
- Pre-training on Synthetic Driving Data for Trajectory Prediction
- HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction
- DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
- Enhancing Mapless Trajectory Prediction Through Knowledge Distillation
- LaKD: Length-agnostic Knowledge Distillation for Trajectory Prediction with Any Length Observations
- Improved Knowledge Distillation via Teacher Assistant
- Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
- CARLA: An Open Urban Driving Simulator
- Argoverse: 3D Tracking and Forecasting With Rich Maps