Distilling Rotation-Averaged Multi-Modal Forecasters into Single-Pass Recurrent Students

Motion forecasters for self-driving and crowd navigation have to do three expensive things. They must give the same answer when the scene is rotated. They must output several distinct possible futures rather than one blurred average. And they must run in milliseconds. The textbook way to get the first property is group averaging: run the model on many rotated copies of the input and average the results. That multiplies inference cost by the number of rotations. It also hides a trap. Averaging the coordinates of multi-modal predictions merges the distinct futures into the same infeasible mean path that multi-modal forecasting exists to avoid. We propose to pay the averaging cost only at training time, inside a teacher model, and then compress the result into a small student that runs once per scene.

The averaging is done correctly for distributions. The teacher's predicted mixtures from a handful of randomly rotated inputs are rotated back and pooled into one larger mixture, so no coordinates are averaged. The student is a linear recurrent network: a recurrent model whose state update has no nonlinearity, so training over time can run in parallel rather than step by step. It decodes all future time steps at once. The student learns from the teacher through a relational loss that compares sets of predicted futures by their mutual geometry, rather than matching futures one by one. Two choices give a mode-by-mode match away: which student future is "mode 3" is arbitrary, and so is the global heading.

Core Claim

A multi-modal forecast is a set of $K$ weighted trajectories. That set carries two symmetries that point-wise distillation handles badly. Rotating the scene rotates every trajectory. Relabelling the modes permutes the set, a permutation group of order $K!$. Index-matched distillation must first fix both. It fixes the rotation with a canonical heading and the labelling with a hard assignment such as Hungarian matching. Both are single-frame canonicalizations of the kind that Equivariant Frames and the Impossibility of Continuous Canonicalization proves must be discontinuous. Our claim is that a loss built from rotation- and order-invariant relations among predicted futures avoids both discontinuities. Such a loss combines a soft transport cost between mode sets in the agent-relative frame with the matching of normalized pairwise distances between modes. With that loss, a single-pass recurrent student can inherit the robustness of a rotation-averaged teacher at roughly the cost of one forward pass. The rotation averaging itself only needs a logarithmic number of sampled rotations.

Concretely, let $p_T(y \mid x)$ be the teacher's mixture over future trajectories $y$ given scene $x$, and let $g_1,\dots,g_m$ be rotations drawn uniformly from a discretized rotation group $C_N$ of $N$ angles. The averaged teacher is

$$\bar p_T(y \mid x) = \frac{1}{m}\sum_{i=1}^{m} p_T\!\left(g_i y \mid g_i x\right),$$

a mixture with $mK$ components. Here $p_T(g_i y \mid g_i x)$ means the teacher is run on the rotated scene and its predicted trajectories are rotated back. The student outputs a $K$-component mixture $q_S$ with modes $\{s_k, \pi_k\}$, where $\pi_k$ is the weight of mode $k$. It is trained with

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \alpha\, \mathrm{OT}_\varepsilon\!\left(q_S, \bar p_T\right) + \beta \left\lVert \frac{D(S)}{\mu(S)} - \frac{D(T)}{\mu(T)} \right\rVert^2 .$$

In this loss, $\mathcal{L}_{\text{task}}$ is the usual ground-truth loss: a winner-take-all regression on the closest mode plus mode classification. $\mathrm{OT}_\varepsilon$ is an entropy-regularized transport cost between the two weighted trajectory sets. The vector $D(\cdot)$ is the sorted list of pairwise endpoint distances among a mode set, so it describes how spread out the possible futures are without depending on mode order or heading. The scalar $\mu(\cdot)$ is its mean, which removes scale as in distance-wise relational distillation. For the teacher, $D(T)$ is computed from the $K$ highest-weight components of $\bar p_T$.

Where This Sits

The proposal combines four bodies of work, each in a different role.

The distillation mechanism comes from Relational Knowledge Distillation. That paper matches mean-normalized pairwise distances and triplet angles between embeddings of different samples, and a ResNet-18 student reached Recall@1 of 82.50% on Cars 196 against its teacher's 77.17%. We apply the same distance relation among the output modes of one prediction, not among samples in a batch. With $K=6$ modes, the quadratic cost the paper flags is negligible.

Beyond Point Predictions: Manifold Expansion and Dual Alignment for Robust Time Series Distillation does two things that matter here. It shows that point-wise distillation of forecasters transfers teacher errors to the student. It also pairs similarity-preserving distillation with optimal transport to distill Transformers into linear students, reaching up to 15.7× speedups. That motivates us and supplies the transport term. However, that work is confined to one-dimensional channel-independent series and has no spatial symmetry or multi-modal output.

The symmetry side has two parts. Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models shows that input-output group averaging over a black-box model, here an LSTM, raises accuracy on the SCAN "Add jump" split from 0% to about 98%, at an $O(|G|)$ inference cost. We keep its averaging but move it to training time. Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry shows that about $O(\log|G|)$ random group elements give near-full approximate symmetry. Its experiment found that 32 elements suffice for $|G|=2^{20}$. That justifies choosing $m$ small. The result is proved only for finite groups under an $L^2$ criterion, so we treat its extension to discretized planar rotations as an empirical question, not an inherited guarantee. The impossibility result on continuous canonicalization constrains the design: it rules out the shortcut of rotating every scene to a fixed heading and claiming exactness.

On the forecasting side, three papers use or test the sequential setting. Trajectory Prediction using Equivariant Continuous Convolution (ECCO) builds exact rotation equivariance into the architecture but decodes autoregressively and deterministically, and names both as open problems. Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data is multi-modal but uses step-by-step LSTMs and agent-centric frames. Hazy Pedestrian Trajectory Prediction via Physical Priors and Graph-Mamba shows that state-space sequence models reach 494.5 FPS on edge hardware but ignores rotation symmetry.

HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction is the closest system. It obtains rotation invariance from local agent-centric frames, avoiding per-agent scene re-normalization, and outputs Laplace mixtures. It neither distills nor removes attention. None of these papers distills a symmetry-averaged multi-modal teacher, and none treats mode order as a symmetry that distillation must respect.

The student's recurrent core follows Resurrecting Recurrent Neural Networks for Long Sequences, which introduces the Linear Recurrent Unit (LRU). The LRU is a diagonal complex-valued linear recurrence trained with a parallel scan in $O(\log L)$ depth. The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity shows that such models cannot track long permutation compositions in one pass. We take that as a constraint that does not bite here. Encoding a 20–50 step kinematic history needs smooth temporal integration, not state tracking. We sidestep the tension that nonlinear mode switching would break associativity by keeping mode generation outside the recurrence: $K$ learned mode queries read the final state and emit all future steps in parallel.

Research Program

We will build two teachers. The first is a high-capacity global-frame Transformer forecaster with no built-in rotation handling. It is the case where averaging should matter most. The second is HiVT, whose invariance is architectural, so that we can tell whether averaging adds anything on top of built-in invariance. Both teachers are wrapped in the distributional averaging above, with $C_N$ at $N=64$ angles and $m$ swept from 1 to 32.

The student encodes each agent's history and each lane polyline with LRU layers. It exchanges information between agents through one round of message passing on relative positions and headings. It then decodes six modes non-autoregressively. Its size is targeted at around the 129.8K parameters ECCO reports, which is well under HiVT-64's 662K. The student sees each scene once, in the raw global frame, with no rotation augmentation in the main condition.

The analysis targets four things. The first is how the student's rotation consistency scales with the teacher's $m$, to test whether the logarithmic sample count transfers to rotations. The second is whether the relational term, the transport term, or both carry the benefit. The third is how gaps behave on agents whose heading is ill-defined: stopped vehicles, and pedestrians moving slower than 0.5 m/s. This is where canonicalization should break. The fourth is wall-clock cost on GPU and on an edge device.

Baselines

The naive reference is the same student architecture trained only on ground truth with the winner-take-all loss, with no teacher and no augmentation. It gives the floor for both accuracy and rotation consistency. A second reference adds random-rotation augmentation to that student under the same number of gradient steps. On the strength of ECCO's results we expect this reference to be weaker than built-in symmetry: there, a data-augmented continuous convolution reached ADE 1.77 against ECCO's 1.62 on Argoverse.

The strongest existing method is HiVT, retrained on our splits with its published configuration and given the same training budget as the teacher. It is the strongest competitor because it already combines invariance, multiple modes and low latency (about 20 ms reported). Lacuna gives no reported minADE or minFDE value for HiVT, nor for SEPT, whose summaries give leaderboard ranks only. Our comparison therefore rests on our own reruns. The only Argoverse numbers in the file are single-trajectory ADE figures: ECCO 1.62 (Trajectory Prediction using Equivariant Continuous Convolution), and LSTM ED+social+map at ADE 2.25 and FDE 4.67 (Argoverse: 3D Tracking and Forecasting With Rich Maps). We will report these alongside ours as context, not as like-for-like competitors to six-mode metrics. On ETH/UCY there is no reported per-split result for Trajectron++ on Lacuna, so we will rerun it.

Beating HiVT means one of two outcomes at equal or lower latency. The first is matching its minFDE at $K=6$ within run-to-run noise while using clearly fewer parameters and running faster on the edge device. The second is beating it on rotation consistency and on the ill-defined-heading slice without losing more than noise on overall minFDE.

Two ablations isolate the key ingredient. The first replaces our relational and transport terms with index-matched point-wise distillation after Hungarian assignment of student modes to the teacher's top-$K$ modes. This is the natural point-wise competitor. The second keeps our loss but uses an unaveraged teacher ($m=1$). Together they separate "respecting mode-order symmetry in the loss" from "averaging the teacher over rotations." A third control averages teacher coordinates rather than distributions, to measure the averaging trap directly. All distilled variants share data, student architecture, optimizer and step count. The distillation weights $\alpha,\beta$ get the same 20-trial tuning budget as the augmentation strength of the augmented reference.

Why not just…?

Why not just canonicalize every scene to the target agent's heading? This is the standard practice and the most serious threat; for moving vehicles it may be as good as anything we propose. The topological result in Equivariant Frames and the Impossibility of Continuous Canonicalization guarantees discontinuity only near inputs where the frame is ambiguous. In driving data those are stopped or slow agents, and that is where the result predicts failure. We will therefore run a heading-canonicalized student with otherwise identical training and report it per speed stratum. We will also measure how sharply predictions jump under small heading noise. If canonicalization matches our averaged-teacher student on every stratum, we will drop the averaging claim and keep only the mode-relational distillation result.

Why not just train the student with rotation augmentation and skip the teacher? Augmentation teaches invariance only as well as the student's capacity allows. In ECCO's comparison the augmented model was both larger and worse than the equivariant one (ADE 1.77 versus 1.62). Our augmented-student baseline tests this directly at matched compute. If it closes the gap, the teacher is unnecessary and we will say so.

Why not just distill point-wise with Hungarian matching? Matching is a hard choice of mode labelling, so the target for each student mode can flip between near-tied teacher modes from one step to the next. Time-series distillation already shows point-wise targets transfer teacher errors (Beyond Point Predictions: Manifold Expansion and Dual Alignment for Robust Time Series Distillation). The Hungarian ablation measures this cost in our setting, including how often assignments flip during training.

Why not just average the student over a few rotations at inference? The approximate-symmetry result suggests a handful of rotations might suffice, but that still multiplies latency by $m$. We include this as a cost–accuracy curve. If it reaches the distilled student's consistency with $m\le2$, the distillation buys little.

Proposed Experiments

The main benchmark is Argoverse 1: 2 seconds of history, 3 seconds of future, minADE, minFDE and miss rate at $K=6$. Argoverse 2 with its 6-second horizon tests whether the student's parallel encoder holds up over longer sequences. ETH/UCY under leave-one-scene-out tests pedestrians, where headings are least reliable.

Beyond the standard metrics we will report a rotation-consistency score. For 64 rotations of each validation scene, we rotate the predicted mixture back and measure the transport distance to the prediction on the unrotated scene. We also report latency and parameter count on one GPU and one edge device.

The decisive checks are three. First, at equal student size, the averaged-teacher relational student must beat both the augmented student and the Hungarian point-wise student on minFDE or rotation consistency. If it does not, the core claim fails. Second, consistency should saturate as $m$ grows, roughly at the logarithmic rate the finite-group result predicts. That would mean saturating near $m\approx6$ for $N=64$. A linear need in $m$ would mean the shortcut does not carry over to rotations. Third, coordinate-averaged teachers should yield students with higher miss rate than distribution-averaged teachers, especially at intersections. That would confirm that averaging must happen over distributions, not over trajectory coordinates.