Models that forecast how cars, pedestrians, or traffic signals affect one another usually pass messages between agents over a graph. The weights on that graph come from attention: a learned score says how much each agent should listen to each other agent. The graph-learning literature has a well-documented failure called over-smoothing. When many rounds of message passing are stacked, every node's representation drifts toward the same vector, and the model loses the ability to tell nodes apart. Demystifying Oversmoothing in Attention-Based Graph Neural Networks proves that softmax attention cannot prevent this. Softmax gives every existing edge a strictly positive weight, so it can never disconnect the graph, and representations still collapse exponentially toward a single direction. Multi-agent forecasting and coordination models are built on exactly this kind of softmax attention, yet nobody has measured whether they collapse. Their designers keep interaction modules shallow, as in CRAT-Pred, without testing whether depth is harmful or merely unnecessary.
This note proposes two things. The first is a diagnostic: measure representation collapse in multi-agent interaction models, calibrated against how much diversity the agents' true future motions actually contain. The second is a mechanism: an attention gate that sets weights to exactly zero whenever two agents' predicted paths never come close within a time horizon. Unlike pruning based on feature similarity, the gate cuts edges by physical reasoning, so agents who must coordinate keep their link. Because it creates real zeros, it breaks the positivity condition behind the collapse theorem in a way that can be tested. The theory makes a sharp prediction. Collapse can only proceed within groups of agents that are physically interacting, so the effective rank of agent embeddings should stay above roughly the number of such groups, however deep the stack.
Core Claim
Over-smoothing in physical multi-agent graphs is not automatically a pathology. Two cars in a platoon should end up with similar representations, because their futures are similar. The right question is whether a model's agent embeddings lose more diversity than the agents' ground-truth futures warrant.
We define the rank deficit of a layer as the ratio $\rho^{(\ell)} = \mathrm{Erank}(\mathbf{H}^{(\ell)}) / \mathrm{Erank}(\mathbf{Y})$. Here $\mathbf{H}^{(\ell)}$ is the matrix of agent embeddings after layer $\ell$ in one scene, and $\mathbf{Y}$ is the matrix of those agents' flattened ground-truth future displacements. Erank is the entropy-based effective rank of Are we measuring oversmoothing in graph neural networks correctly?. That paper reports that Erank tracks accuracy on node classification (average Pearson 0.91) where Dirichlet energy does not. It also notes that high rank alone does not guarantee useful features, which is exactly the problem that normalizing by the rank of $\mathbf{Y}$ is meant to address.
The claim is falsifiable in two ways. It is supported if, in regimes where depth helps (network-level traffic control, and dense scenes where effects travel through chains of agents), $\rho$ falls well below 1 as depth grows, and a kinematic gate both keeps $\rho$ near 1 and turns extra depth into accuracy. It fails if $\rho$ stays near 1 in ungated models, or if a gate with no physics does equally well.
Mechanism
At each interaction layer $\ell$, a light head decodes a provisional future trajectory for every agent. This reuses the intermediate-endpoint idea of Social-PatteRNN, which conditions attention on predicted sub-goals rather than current positions. For every pair of agents, we compute the time of closest approach $t^*_{ij}$ and the distance at that moment $d^*_{ij}$ from their provisional trajectories. The attention weights then become
$$\alpha_{ij}^{(\ell)} = \frac{g_{ij}^{(\ell)} \exp(e_{ij}^{(\ell)})}{\sum_k g_{ik}^{(\ell)} \exp(e_{ik}^{(\ell)})}, \qquad g_{ij}^{(\ell)} = \mathbb{1}\left[t^*_{ij} < \tau,\ d^*_{ij} < \delta\right],$$
In this expression, $e_{ij}$ is the usual learned attention score and $g_{ij}\in\{0,1\}$ is the gate. The threshold $\tau$ is a horizon in seconds and $\delta$ is a clearance distance in meters. For signal control, the gate uses the travel time of predicted queue fronts between intersections in place of the collision geometry. The gate is anisotropic and asymmetric by construction. A slower car ahead on your path passes the gate, while a car diverging behind you does not. In this respect it follows the heading- and bearing-dependent influence zones of Sekhon et al.. By contrast, the isotropic gravity decay in Gravityformer, which regularizes attention by $m_i m_j / r^\beta$ over fixed city zones, cannot express this.
Gradients pass to $e_{ij}$ normally. The gate itself is treated as a stop-gradient during training, and $\tau$ and $\delta$ are swept rather than learned, so that the gate cannot be quietly relaxed into dense attention.
The theoretical point needs care, because the result of Wu et al. is not simply "zeros fix it." Products of row-stochastic matrices still drive the embeddings toward consensus when the union of graphs over many layers stays connected. We therefore extend their joint-spectral-radius argument to the gated, layer-varying case. The target statement is that collapse is confined to connected components of the jointly gated graph. The embedding matrix then converges to a subspace whose dimension is at least the number of such components $c$, which gives the predicted floor $\mathrm{Erank}(\mathbf{H}^{(\ell)}) \gtrsim c$.
Research Program
Phase one is purely diagnostic. We instrument four existing architectures and sweep interaction depth from 1 to 32 layers, logging Erank, numerical rank, Dirichlet energy, and $\rho$ at every layer. The four are the actor-to-actor stage of LaneGCN, the attention module of CRAT-Pred, the stacked graph attention layers of CoLight, and the soft coordination graph of DICG. We stratify scenes by agent count, since the dispersion of attention in crowds is the suspected driver of collapse.
Phase two adds the kinematic gate and runs the theory check. We count the components $c$ of the jointly gated graph per scene and plot measured Erank against $c$.
Phase three asks whether preserving rank turns depth into accuracy, and in which regime. We expect a split. In Argoverse vehicle scenes, most interactions are local and depth probably adds little. On the 196-intersection Manhattan network, CoLight's premise is that multi-hop propagation of congestion matters.
Baselines
The naive reference point is the shallow ungated model at its best depth: one or two interaction layers, as CRAT-Pred and GATraj ship them. Every deep variant must beat this model at its own best configuration, not merely beat deep ungated models. On Argoverse, the strongest cited numbers are from LaneGCN, which reports test minFDE of 1.36 m and miss rate 0.16 at K=6. CRAT-Pred reports a map-free test minADE of 1.82 m with about 515k parameters. Because we modify interaction depth, we will retrain LaneGCN from its published configuration on the validation split under our own budget and compare against that reproduction. The published test number serves only as a sanity anchor.
On CityFlow, CoLight reports an 11.69% travel-time improvement over the next-best method on real networks. No absolute travel-time figure was found on Lacuna for CoLight, so we rerun CoLight ourselves. On Traffic Junction Hard, DICG reports an 82.2% success rate.
The strongest competitors from the over-smoothing literature are DYNAMO-GAT, which prunes attention edges between nodes whose features become too correlated, and PairNorm, which re-centers and rescales features to keep their total pairwise distance fixed. Both are drop-in rank preservers with no physical knowledge. No reported result was found on Lacuna for either method on any trajectory or traffic-control benchmark. We also include the high-pass diversification channel of FB-GNN as a frequency-based alternative, and a learned sparse-attention baseline that uses sparsemax in place of softmax.
The ablation that isolates the key ingredient is a random gate. It zeros the same fraction of edges per scene as the kinematic gate, but chooses which edges at random. If it matches the kinematic gate, the benefit comes from sparsity alone and not from physics.
For fairness, all methods share data splits, backbone, parameter count within 5%, and a fixed hyperparameter search of 20 trials per method, run on identical GPU budgets. Beating a baseline means a gain larger than the across-seed standard deviation over five seeds on the primary metric (minFDE and miss rate, average travel time, or success rate), obtained at the method's best depth.
Why not just…?
Why not just keep the interaction module shallow? This is the most serious threat, and for Argoverse it may well be correct. The file's evidence that deep interaction is needed is indirect. LaneGCN's dilated convolutions act on the lane graph, not on agent-to-agent links. CoLight and DICG argue for multi-hop reasoning, but they do not show a depth sweep. We do not argue around this. Phase three reports the best-depth comparison per regime. If the shallow model wins everywhere, the diagnostic result still stands, and the conclusion is that collapse is not what limits these models. We would regard that as a useful negative result.
Why not just use DYNAMO-GAT's correlation-based pruning? DYNAMO-GAT cuts edges between nodes whose features are highly correlated. In traffic, the agents with the most correlated features are often the ones sharing a lane or a crosswalk, and these are precisely the pairs that must keep talking. Our prediction is that DYNAMO-GAT preserves rank but raises the collision rate in dense scenes. We will measure predicted-trajectory overlap between agents on SDD and ETH/UCY to check this directly.
Why not just use sparsemax or top-k attention, which also produce exact zeros? This is a real threat to the physics claim. Learned sparsity also breaks softmax positivity, so it might preserve rank just as well. The comparison is direct: kinematic gate against sparsemax against the random gate, all at matched sparsity. If sparsemax matches the gate on both $\rho$ and accuracy, we will report that the physical prior is unnecessary and that only exact zeros matter.
Why not just add residual connections and LayerNorm? Zhang et al. report that normalization changes feature magnitude without stopping rank collapse. PairNorm reports that residual connections still degrade with depth. These results come from citation graphs, however, so we include residual connections plus LayerNorm as a baseline at every depth rather than assuming the result transfers.
Proposed Experiments
We use four benchmarks: Argoverse motion forecasting (minADE, minFDE, and miss rate at K=6), ETH/UCY and SDD (ADE and FDE, plus the collision rate between predicted agent trajectories), CityFlow Hangzhou, Jinan, and the 196-intersection Manhattan network (average travel time), and Traffic Junction Hard (success rate).
Three checks decide the outcome. The first tests collapse. Ungated deep models must show $\rho$ falling well below 1 with depth, and more steeply in crowded scenes. If not, the premise fails. The second tests the rank floor. Under the gate, per-scene Erank should track the component count $c$ and should not fall below it as depth grows. A violation would mean the extended theory is wrong. The third tests conversion into accuracy. On Manhattan CityFlow and Traffic Junction Hard, the gated model at its best depth must beat both the shallow model and the reproduced CoLight or DICG, and it must also beat the sparsemax and random-gate variants. Only if all three checks hold does the proposal's specific claim survive: physically motivated exact zeros, not merely sparsity or depth, are what let multi-agent interaction models benefit from depth.
References
- Demystifying Oversmoothing in Attention-Based Graph Neural Networks
- CRAT-Pred: Vehicle Trajectory Prediction with Crystal Graph Convolutional Neural Networks and Multi-Head Self-Attention
- Are we measuring oversmoothing in graph neural networks correctly?
- Social-PatteRNN: Socially-Aware Trajectory Prediction Guided by Motion Patterns
- A Spatially and Temporally Attentive Joint Trajectory Prediction Framework for Modeling Vessel Intent
- A Gravity-informed Spatiotemporal Transformer for Human Activity Intensity Prediction
- Learning Lane Graph Representations for Motion Forecasting
- CoLight: Learning Network-level Cooperation for Traffic Signal Control
- Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
- A Dynamical Systems-Inspired Pruning Strategy for Addressing Oversmoothing in Graph Attention Networks
- PairNorm: Tackling Oversmoothing in GNNs
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks