Right-of-Way as a Fitness Contest: An Identifiable Latent for Who Yields in Multi-Agent Forecasting

When two cars reach a merge at the same time, one of them yields. Which one yields is often not visible in either car's recent motion. It depends on something persistent about the driver: how assertive they are, how urgent their trip is, how big their vehicle is. Network science has a name for this kind of persistent, unobserved advantage. It calls it intrinsic fitness: a per-node quality that decides who wins new connections, independent of how many connections the node already has. We propose to give every agent in a traffic or pedestrian scene one scalar latent "assertiveness" and to let the forecasting model use it in only one place, the odds of winning each pairwise right-of-way contest. Because the same agent's assertiveness must explain all of its contests with different partners, the latent is statistically identifiable whenever the scene's contests form a connected graph. That identifiability is the property recent theory ties to preventing posterior collapse, the failure in which a generative model learns to ignore its latent variables.

A second consequence follows. Who yielded is a future event, so the latent can only be inferred well by looking at the whole scene in hindsight. This gives a concrete multi-agent reason to use bidirectional inference during training: the model pools evidence across time and across every partner an agent negotiates with. At test time a causal network, one that sees only the past, must predict the latent from history. The payoff we expect is not mainly lower average displacement error. We expect coherent joint futures, in which an agent sampled as assertive passes every conflicting partner in that sample, and better accuracy on the question a planner actually needs answered: who goes first.

Core Claim

Social posterior collapse in multi-agent variational autoencoders has a structural cause. The decoder can explain each agent's future from that agent's own momentum, so no latent is needed. The standard remedy is extra machinery. We argue the remedy should be structural instead: a latent that enters the decoder only through pairwise comparisons, the way fitness enters attachment probabilities in growing networks.

For each pair of agents $i,j$ whose paths conflict in a scene, let $c_{ij}\in\{0,1\}$ indicate whether $i$ reaches the conflict point first. We model

$$ p(c_{ij}=1 \mid \eta, x) = \sigma\big(g_\theta(x_{ij}) + \eta_i - \eta_j\big), $$

where $\eta_i$ is agent $i$'s scalar assertiveness latent, $x_{ij}$ is the observed pairwise kinematic and map context (gaps, speeds, lane priority), $g_\theta$ is a flexible network, and $\sigma$ is the logistic function. Trajectories are then decoded conditioned on the sampled contest outcomes together with per-agent motion latents. The role of $g_\theta$ mirrors the degree-dependent attachment kernel in PAFit, a nonparametric maximum-likelihood method for measuring preferential attachment in temporal networks. The role of $\eta$ mirrors node fitness in Joint estimation of preferential attachment and node fitness, which separates a node's "rich-get-richer" advantage from its intrinsic quality.

The separability argument transfers directly from those papers. Fitness is recoverable because it is constant across many attachment events. Assertiveness is recoverable because it is constant across many contests. The pairwise-difference form is identifiable up to an additive constant exactly when the contest graph, meaning agents joined by conflicts, is connected; this is the familiar condition for comparison models of this form.

We then invoke the theorem in Posterior Collapse and Latent Variable Non-identifiability, which shows that collapse occurs if and only if latents are non-identifiable. It predicts that $\eta$ should stay informative where contests exist, and should correctly fall back to its prior for isolated agents. That paper achieves identifiability with invertible decoders built from input-convex networks, known as Brenier maps. Those decoders cost second-order gradients. Our construction obtains identifiability from the likelihood's structure at ordinary first-order cost.

How This Differs From Existing Bridges

Three papers sit closest to this idea, and each supplies a piece we reuse or react to.

Exploring Social Posterior Collapse identified the failure itself. It showed that a vanilla VAE attends to about 8.68% of neighbors on INTERACTION highway merging. It then countered this with an auxiliary future-blind decoder and sparse entmax attention, raising the share to 15.5%. It motivates our work. We differ in making social dependence a property of where the latent is allowed to enter the model, not of an added loss. We also note its admitted weakness on ETH/UCY pedestrians, which we treat as a test case rather than something to explain away.

SocialVAE supplies the method for hindsight inference. It uses a backward-running recurrent network over each agent's ground-truth future to shape per-timestep latents. Its hindsight is per-agent, however, and its social attention is purely geometric. Our posterior over $\eta_i$ instead pools contest evidence across all of $i$'s partners and the full horizon, which is a minimal instance of the spatio-temporal graph smoothing the file notes is missing.

Mind the Gap supplies the argument for why this matters. It proves that an inference network restricted to the past approximates a product of posteriors across futures rather than their mixture. A history-only posterior over assertiveness would therefore be overconfident and badly placed exactly when the decisive evidence, the yield itself, lies in the future.

SocialCVAE is a related but different design. It conditions on physics-style repulsion energy maps whose parameters are the same for every agent and which are expensive to compute on a grid. Our heterogeneity is per-agent and costs one scalar per agent.

Research Program

We would build the model in three parts. The first is a scene encoder, a standard graph-attention encoder over history and map, which proposes conflicting pairs from map lane intersections on vehicle data and from future path proximity on pedestrian data. The second is a contest layer implementing the equation above. The third is a trajectory decoder conditioned on contest outcomes and per-agent motion latents.

Training maximizes an evidence lower bound. In it, the inference network for $\eta$ is a hindsight smoother that reads the full scene, including observed contest outcomes $c_{ij}$ derived from ground-truth futures. The prior for $\eta$ is a causal network reading history only. The KL term therefore teaches the causal network to predict assertiveness from behavioral cues such as gap acceptance and acceleration profiles. This division directly addresses the deployment gap between training with the future and running without it.

We would first validate on a controlled simulator: intersection and merge scenes with planted $\eta$ values and tunable contest-graph connectivity. Here we can check recovery of $\eta$, and check whether the information carried by the latent tracks connectivity as the non-identifiability theorem predicts.

We would then move to real data. On INTERACTION (Zhan et al.), we would use its interactive vehicle pairs with time-to-conflict at or below 1 s as the primary evaluation. Argoverse and ETH/UCY provide comparison with published numbers. SportVU NBA, where the same players negotiate repeatedly over long clips, is the regime most favorable to persistent fitness.

Baselines

The naive reference is a marginal predictor. It is the same encoder and decoder with no contest layer and no $\eta$, trained as a standard conditional VAE, and it samples each agent's future independently. It shows what the scene encoder alone achieves and gives the floor for joint-consistency metrics. A method only counts as useful if it clearly improves yield-order accuracy over this floor.

The strongest existing competitor on the collapse question is Social-CVAE from Tang et al.. It is the only method in the file designed specifically against social collapse, and it reports INTERACTION Generalization-track FDE of 2.70 (versus 3.85 for a standard VAE) and Argoverse minADE 1.59 and minFDE 3.52 (versus 1.86 and 4.17). We would reimplement it on our shared encoder with identical data splits, horizons, and hyperparameter search budget, and also report its numbers under its original protocol. Our reproduction may not match the paper, since the file does not give full protocol details, so we would state the gap explicitly. Beating it means lower FDE on the Generalization track and higher yield-order accuracy on critical pairs, with confidence intervals over seeds that exclude zero.

On pedestrians and NBA, the strongest competitor is SocialVAE, including its final-position clustering. It reports a 12% ADE improvement over prior methods on ETH/UCY, rising to 21% ADE and 30% FDE with clustering. Our file contains only these relative figures and no absolute values: no reported absolute ADE/FDE was found on Lacuna for SocialVAE. We would therefore run the released configuration under the standard 8-frame observation and 12-frame prediction protocol with 20 samples, and compare on identical splits.

Two ablations isolate the key ingredients. The first keeps the contest layer but infers $\eta$ with the causal network only, removing hindsight; this tests the smoothing claim. The second replaces the pairwise-difference structure with an unconstrained per-agent latent fed to the decoder everywhere; this tests the identifiability-by-structure claim, and we predict it will collapse back toward ego momentum.

Why not just…?

Why not just feed observable proxies like vehicle class and size? Class and size capture part of fitness, and INTERACTION provides some of it. We would include these observables inside $g_\theta$ in every variant, so $\eta$ can only absorb the residual persistent tendency. If the learned $\eta$ adds nothing to yield-order accuracy once class and size are in $g_\theta$, the latent is unnecessary and we would report that. The simulator guarantees a positive control where hidden assertiveness exists by construction.

Why not just use KL annealing or free bits on an ordinary social CVAE? Tang et al. report that even a socially conditioned vanilla VAE attends to only about 8.68% of neighbors. The non-identifiability result in Wang et al. implies that schedules delay collapse rather than removing its cause. This remains an empirical question in trajectories, so we would run an annealed, free-bits version of the unconstrained-latent ablation with the same tuning budget, and measure latent mutual information and Agent Ratio. If annealing matches our information retention and yield accuracy, the structural argument loses its practical force.

Why not just learn a deterministic assertiveness score from history, with no latent and no hindsight? This is the most serious threat. A discriminative head with the same pairwise structure, trained directly on observed $c_{ij}$ outcomes, might learn the same cues. Our claimed advantage is twofold. First, samples in which a shared stochastic $\eta_i$ couples all of $i$'s contests should be jointly coherent, while a deterministic score yields marginal contest probabilities that are sampled independently. Second, hindsight pooling should give the causal prior a cleaner training target than individual noisy contest labels. We would include this deterministic-score model as a baseline. If it matches on both yield accuracy and joint coherence, the honest conclusion is that the pairwise fitness structure is the contribution and the variational machinery is not.

Why not just use a diffusion model for joint trajectories? Joint diffusion models such as Masked Trajectory Diffusion can produce coherent scenes. However, they expose no interpretable yield variable and need many denoising steps per sample. The contest layer is orthogonal and could condition such a model. We would treat diffusion as future work, not as a competitor we claim to beat.

Decisive Checks

The first check is on the simulator. Posterior mutual information for $\eta_i$ should rise with the number of distinct contest partners of agent $i$ and be near zero for isolated agents. The unconstrained-latent ablation should show no such relationship. Failing this refutes the identifiability mechanism.

The second check uses the INTERACTION critical pairs. Yield-order accuracy, meaning which agent reaches the conflict point first in the best-of-$K$ joint sample, should improve over both Social-CVAE and the deterministic-score model. The hindsight ablation should sit in between.

The third check measures joint coherence using the critical-interaction evaluation style of Mode Collapse Happens. It should show fewer samples with contradictory contest outcomes, such as $i$ beating $j$, $j$ beating $k$, and $k$ beating $i$ in one sample.

Finally, on ETH/UCY, where Tang et al.'s remedy weakened, we expect gains only in dense scenes with connected contest graphs. Finding gains in sparse scenes would itself suggest that the model is exploiting something other than fitness.