Forecasting where every car, cyclist and pedestrian in a scene will move requires modeling how agents influence one another. The most accurate way to keep such models independent of the scene's orientation is to describe every pair of agents by its relative distance and angle, as GoRela does. That design has an obvious problem: its cost grows with the square of the number of agents. The cheap alternative is to route interactions through a few shared summary tokens ("hubs"), as the factor-based transformer f-MAT does for control. This makes cost linear, but existing hub designs are blind to geometry. Rotate the scene and their predictions change, unless you train them on many rotated copies of the data. This proposal asks what it costs, in accuracy and compute, to keep rotation symmetry once interactions are compressed through hubs. It also asks whether a small, deliberately measured violation of that symmetry is the cheapest way to close the gap.
The idea in two sentences: each hub stores what its member agents are doing as a short list of rotation-aware frequency components. Each agent reads that summary in its own heading frame, so the whole model stays exactly rotation-invariant at linear cost. Truncating that list loses fine angular detail. We therefore add a cheap channel that is not symmetric and is penalized to stay nearly symmetric, and we measure where on the accuracy–compute–symmetry trade-off this beats simply keeping more frequencies. The hubs also carry the discrete choice of joint future for their members, so agents that negotiate the same space choose their futures together rather than independently.
Core Claim
Exact rotation invariance and sub-quadratic interaction cost are compatible. A hub can pool its members' relative positions and headings into circular Fourier coefficients: a hub-level summary in which the $\ell$-th component rotates by a known phase when the scene rotates. Pooling and reading these coefficients costs $O(NmL)$ for $N$ agents, $m$ hubs and $L$ frequencies. The hard part is choosing $L$. Low orders lose the angular structure that matters most, for example four vehicles converging on an intersection from four directions, where first-order summaries cancel out. High orders cost compute.
Our central claim is that a low-order steerable hub plus a small non-equivariant residual, kept near-equivariant by a soft penalty, matches the accuracy of exact pairwise-relative models at linear cost. It also dominates both extremes at equal compute: purely exact hubs with larger $L$, and purely geometry-blind hubs trained with augmentation. This is a falsifiable claim about a frontier, not a claim that relaxation always helps. If exact hubs with larger $L$ reach the same frontier, relaxation is unnecessary, and the experiments below are built to detect that.
Where This Sits
Three lines of prior work each supply one piece and leave the others open.
Exactly invariant forecasters motivate the geometric prior. ECCO enforces rotation equivariance with continuous convolutions and reaches ADE 1.62 on Argoverse, against 1.77 for an augmented non-equivariant convolution, with about 8x fewer parameters. It is deterministic, however, and its convolutions are dense. GoRela obtains strong sample efficiency from pairwise relative encodings, but those encodings are per-pair by construction. HiVT avoids pairwise cost with agent-centric local frames inside a fixed radius, followed by a global stage. Its symmetry is exact, and its interactions are truncated by a hand-set radius rather than learned groupings. We keep their exact invariance as the default and change how interactions are routed.
Linear-cost interaction models supply the routing. f-MAT replaces all-to-all attention with a bipartite graph of agents and hubs and reports an 18x speedup over the Multi-Agent Transformer with 144 agents. It uses a static, predefined hub graph, which it names as its main limitation, and no geometric structure. VAIN gets linear cost from per-agent attention vectors but reports failure when every agent strongly affects every other. That result constrains us: hubs must be reassigned dynamically, or dense negotiations will be averaged away. We reuse the bipartite routing and replace its static graph with soft assignments recomputed at every layer.
Approximate-symmetry methods supply the relaxation. Relaxed steerable convolutions break symmetry through coordinate-dependent filter weights on grids. Projection-based regularisation and constrained optimization with learned slack penalize, rather than forbid, equivariance violations. PEnGUiN blends an equivariant and a plain graph network through a learned per-node gate, but only for reinforcement-learning policies on quadratic graphs, with no trajectory distributions. Theory indicates that approximate symmetry is far cheaper to achieve than exact symmetry, which is the hypothesis we test in the compute sense.
The differences from each are concrete. PEnGUiN relaxes symmetry to handle environments whose dynamics are asymmetric. We relax it to buy back expressivity lost to compression, and we measure the violation rather than gating it by an unconstrained MLP. Unlike RSteer, our relaxation lives in agent-to-hub attention, not grid filters.
There is one point where prior work seems to cut against us. When the map is part of the input, rotating the entire scene is an exact symmetry of driving data: lane rules and signals rotate with the scene. So the familiar argument that "roads break symmetry" does not justify relaxation here, and we do not rely on it. The honest justifications are the expressivity cost of truncation, and one genuinely asymmetric factor in logged data. Detection and occlusion quality depend on direction relative to the recording vehicle's sensors, so a residual channel given ego-relative information can represent something no invariant model can.
Research Program
The encoder follows HiVT-style agent-centric local encoding of each agent's history and nearby lane vectors. What we replace is the global interaction stage.
At each layer, $m$ hubs (we will test $m$ from 8 to 64) have centers $c_h$ computed as assignment-weighted means of member positions, which makes them exactly equivariant. Agents are softly assigned to hubs by an invariant function of distance to each center and of their local embeddings, in the manner of slot attention, so hub membership changes as the scene evolves. Let $x_i$ be agent $i$'s position, $\phi_i$ its heading, $\theta_{ih}$ the angle of $x_i - c_h$, $w_{ih}$ its soft assignment, and $s_i$ an invariant feature vector. Hub $h$ stores, for $\ell = 0,\dots,L$,
$$H_h^{(\ell)} = \sum_i w_{ih}\, g_\ell(s_i, \lVert x_i - c_h\rVert)\,\big(e^{\mathrm{i}\ell\theta_{ih}},\ e^{\mathrm{i}\ell\phi_i}\big),$$
where $g_\ell$ is a small network. Rotating the scene by $\beta$ multiplies every $H_h^{(\ell)}$ by $e^{\mathrm{i}\ell\beta}$. Agent $i$ reads the hub as $e^{-\mathrm{i}\ell\phi_i}H_h^{(\ell)}$, which is invariant, so the read-out is exact.
The relaxed residual $r_h$ is a low-dimensional hub vector pooled from the same members, but using absolute coordinates in the ego frame. It is trained under an equivariance-error budget $\varepsilon$:
$$\min_\theta \ \mathcal{L}_{\text{forecast}}(\theta) \quad \text{s.t.} \quad \mathbb{E}_{x,R}\,\lVert f_\theta(Rx) - f_\theta(x)\rVert^2 \le \varepsilon.$$
Here $f_\theta$ is the predicted output, $R$ a random rotation, and the constraint is enforced with a learned dual variable following Manolache et al.. This avoids hand-tuning a penalty weight, which the research file identifies as the main practical weakness of soft-equivariance methods. Sweeping $\varepsilon$ from 0 upward and $L$ from 0 to 8 traces the frontier the core claim is about.
For multiple futures, each hub carries a categorical latent over $K$ joint modes. Each agent's Laplace-mixture decoder is conditioned on the modes of the hubs it belongs to, weighted by assignment. Agents sharing a hub therefore commit to a shared joint future. This targets the joint-inconsistency limitation that SEPT acknowledges, without the step-by-step joint rollouts of Multiple Futures Prediction and without the quadratic space-time attention analyzed by Girgis et al.. Decoding is parallel across hubs, in the spirit of f-MAT's non-autoregressive decoding.
Baselines
The naive reference points come in two forms. The first is constant-velocity extrapolation, which we compute ourselves. The second is the Argoverse 1 nearest-neighbor and LSTM baselines in the Argoverse paper, with reported 3-second ADE 2.25 and FDE 4.67 for the social-plus-map LSTM. These establish the floor.
The strongest exact competitor is a GoRela-style pairwise-relative model: agent-agent edges within perception range, each encoded by relative distance, relative heading and bearing. This is the accuracy target a linear-cost model must match, because its invariance is exact and it loses no information to compression. The file gives no number for it (no reported result found on Lacuna for GoRela's Argoverse 2 BrierMinFDE value). We will therefore reimplement its interaction layer inside our shared encoder and decoder.
The second strong competitor is HiVT with its radius-based local stage and global stage, reimplemented in the same codebase. It is the existing linear-style invariant design, and it needs no learned hubs. No reported numeric minADE for HiVT was found on Lacuna; the file records only its first-place leaderboard rank.
For external calibration on Argoverse 2, the only reported figure in the file is from QML: minADE ($K=6$) of 0.6882 and miss rate 0.1894 on the 2022 challenge test set. We treat that as a reference, not a matched-budget comparison.
All reimplemented models will share the same local encoder, decoder, training data, epoch count and hyperparameter search budget, and they will be matched on either parameter count or measured latency. We report both matchings.
Four ablations isolate the key ingredients. The first is steerable hubs with $\varepsilon = 0$, which is exact and has no residual. The second is geometry-blind f-MAT-style hubs with rotation augmentation. The third is our hubs with a PEnGUiN-style learned gate between the steerable and residual channels in place of the constrained budget. The fourth is our model with independent per-agent modes in place of hub-level joint modes.
Beating the field means one of two things. Either we match the pairwise model's minFDE and miss rate within noise across three seeds while showing linear latency growth where the pairwise model grows quadratically, or we dominate every ablation on the accuracy-versus-latency frontier.
Why not just…?
Why not just train a geometry-blind hub model with random rotations? Augmentation teaches invariance only approximately and consumes capacity. On Argoverse, ECCO's exact equivariance beat an augmented convolution (1.62 vs. 1.77 ADE) with 8x fewer parameters, and GoRela reports stable accuracy under rotation where baselines degrade. Our second ablation measures this directly, including performance on 10% of the training data, where we expect the gap to widen.
Why not just use HiVT's fixed radius plus a global stage? This is a serious competitor, because it is already cheap and exactly invariant. Its weakness should appear in two places. The first is dense clusters, where everything inside the radius interacts at local quadratic cost. The second is interactions just beyond the radius, where a truncated radius misses long-range context. We test it on a density stress set, described below, and on Argoverse 2's interaction-mined scenarios stratified by agent count. If HiVT matches us at every density, the hub design has no reason to exist, and we will say so.
Why not just keep the model exact and raise the number of frequencies? This is the most direct threat, because it needs no relaxation machinery. The $L$-by-$\varepsilon$ sweep settles it. If the frontier traced by exact hubs with larger $L$ coincides with the relaxed frontier at equal latency, the relaxation claim is false. The contribution then reduces to exact steerable hubs, which is still a new linear-cost invariant interaction layer for forecasting.
Why not just predict each agent independently? This is cheap per scored agent, but it re-encodes the scene for every target, and SEPT itself notes that it cannot guarantee joint consistency. The fourth ablation measures predicted inter-agent overlap directly.
Proposed Experiments
The main benchmark is the Argoverse 2 motion forecasting set from Wilson et al., reporting minADE, minFDE, miss rate and Brier-minFDE at $K=6$. We also report joint metrics we compute ourselves: joint minFDE over agents sharing a scene, and the rate of predicted trajectory pairs that come within vehicle footprint distance. Argoverse 1 serves as a secondary check against ECCO's and MFP's reported numbers (minADE 1.399 for MFP).
To stress scaling, we build a density set by composing Argoverse 2 scenes on shared maps, taking $N$ from 30 to 500 agents, and report latency and memory against $N$. Scaling slopes are more informative there than absolute accuracy. NBA SportVU tests off-map, strategically dense interaction; Ivanovic et al. report NLL of −19.00 for their CVAE model, which serves as the reference.
The decisive checks are three. First, the frontier plot described above. Second, a measured equivariance error on held-out rotated scenes that stays within the budget $\varepsilon$. Third, a sensor-asymmetry probe. We compare residual-channel usage in scenes where agents are occluded relative to the ego vehicle against scenes where they are fully observed. If the residual carries information only there, that confirms the one genuine symmetry-breaking factor we identified. If it is used uniformly, it is compensating for truncation, and the frontier result must stand on its own.
References
- GoRela: Go Relative for Viewpoint-Invariant Motion Forecasting
- Towards Efficient Collaboration via Graph Modeling in Reinforcement Learning
- Trajectory Prediction using Equivariant Continuous Convolution
- HiVT: Hierarchical Vector Transformer for Multi-Agent Motion Prediction
- VAIN: Attentional Multi-agent Predictive Modeling
- Approximately Equivariant Networks for Imperfectly Symmetric Dynamics
- Approximate Equivariance via Projection-Based Regularisation
- Learning (Approximately) Equivariant Networks via Constrained Optimization
- PEnGUiN: Partially Equivariant Graph NeUral Networks for Sample Efficient MARL
- Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry
- SEPT: Towards Efficient Scene Representation Learning for Motion Prediction
- Multiple Futures Prediction
- Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction
- Argoverse: 3D Tracking and Forecasting With Rich Maps
- QML for Argoverse 2 Motion Forecasting Challenge
- Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
- Generative Modeling of Multimodal Multi-Human Behavior