Agent-Factorized Importance Weighting for Multi-Agent Trajectory Models

Models that forecast how cars and pedestrians will move often use a variational autoencoder (VAE), a generative model that compresses each scene into a random "latent" code and decodes possible futures from it. These models have a known failure: they learn to ignore the latent code and the neighboring agents. They forecast each agent from its own past, as if it moved alone. A standard way to make such models use their latents is to train with a tighter objective that averages over $K$ sampled codes and weights each one by how well it explains the data. This is importance weighting. A specific gradient estimator, doubly reparameterized gradients (DReG), keeps the training signal for the encoder clean as $K$ grows. This proposal argues that this recipe breaks in the multi-agent setting for a structural reason, and that the fix uses the interaction graph the forecasting model already has.

The reason it breaks is dimensionality. When all $N$ agents share one joint latent, each sample's importance weight is a product of $N$ per-agent ratios. Its noise therefore compounds multiplicatively, so the number of samples needed for averaging to help grows exponentially with the number of agents. Past a handful of agents, a $K$-sample joint objective behaves like the single-sample one, and DReG's weighting puts all mass on one sample. We propose to draw $K$ codes per agent instead of $K$ joint codes. The model's likelihood is then averaged over all $K^N$ cross-agent combinations, computed exactly by message passing on a sparse interaction graph. We also derive a doubly reparameterized gradient for this estimator. In the uncoupled case, the cost of the multiplicative noise drops from exponential in $N$ to linear in $N$. Our core empirical question is whether this restores the benefit of tight objectives for joint forecasting and reduces the tendency to ignore neighbors, measured directly on INTERACTION, Argoverse and ETH/UCY.

Core Claim

Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling named the failure "social posterior collapse": the model drops neighbor information. Its fixes are architectural, namely an auxiliary decoder without access to the future and sparse $\alpha$-entmax attention. It trains with the single-sample objective throughout. The obvious next step is to swap in a $K$-sample importance-weighted objective with DReG, following Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. We expect this to underdeliver. Each agent has a latent of dimension $d_0$, so the joint latent has dimension $N d_0$. On the difficulty of unbiased alpha divergence minimization shows that weight-based estimators accumulate noise multiplicatively across dimensions when the approximation and target disagree. That paper leaves open whether structured factorizations can break this accumulation, and multi-agent scenes come with exactly such a structure.

Write $z_i^k$ for the $k$-th of $K$ codes drawn for agent $i$ from its encoder $q_i$. Write $h$ for the observed history and $x$ for the future trajectories. Assume the model's log-joint decomposes into factors $f_c$ over small groups $c$ of agents, the cliques of the interaction graph. Our estimator is

$$\hat Z = \frac{1}{K^N}\sum_{k_1,\dots,k_N}\ \prod_{c}\ f_c\big(x, h, z_c^{k_c}\big)\ \prod_{i=1}^N \frac{1}{q_i(z_i^{k_i}\mid x,h)},$$

where the sum runs over every choice of one code per agent. Samples are independent across agents, so every term in the sum has expectation $p(x\mid h)$. That makes $\hat Z$ unbiased, and $\mathbb{E}\log\hat Z$ is a valid lower bound on the log-likelihood. The $K^N$-term sum is computed by variable elimination, at cost roughly $N K^{w+1}$ for a graph of treewidth $w$. Sparse attention of the kind used by Social-CVAE yields exactly the low-treewidth graphs this requires.

The scaling argument is simplest when agents are uncoupled. Let $v_i$ be the relative variance of agent $i$'s single-sample weight. Averaging $K$ joint samples gives relative variance $\big(\prod_i(1+v_i)-1\big)/K$. The per-agent combinatorial estimator gives $\prod_i(1+v_i/K)-1 \approx \sum_i v_i/K$. Both forms follow directly from multiplying independent mean-one estimators. The first needs $K$ exponential in $N$; the second needs $K$ linear in $N$. With coupling factors the truth lies between these extremes, and measuring where is part of the program.

The algebraic idea of averaging over all combinations of per-variable samples has precedents in the broader variational literature that are not in our file. Our contribution is to derive its doubly reparameterized gradient, place it on learned and dynamic interaction graphs, and test whether it changes posterior collapse in forecasting.

Research Program

The first piece is theory. We will derive the DReG form of the gradient of $\log\hat Z$ with respect to each agent's encoder parameters. Every $z_i^k$ is reparameterized, so the score-function terms can be rewritten in pathwise form by the identity Tucker et al. use. We expect the single-sample weights $\tilde w_k^2$ of standard DReG to be replaced by squared per-agent "responsibilities", the marginal share of $\hat Z$ that flows through sample $k$ of agent $i$. These come out of the same message-passing pass.

For the learned conditional prior on each agent's code, we will apply the re-expression trick of Generalized Doubly Reparameterized Gradient Estimators. That paper warns that indirect score functions cannot all be removed when prior and posterior factorize differently. We will therefore deliberately give the encoder and prior the same per-agent factorization, each conditioned on the shared encoded history. This is the condition under which the trick is exact. We will verify unbiasedness numerically and prove that the encoder's gradient signal-to-noise ratio (expected gradient divided by its standard deviation) grows with $K$. Rainforth et al. proved this ratio decays as $O(1/\sqrt K)$ for plain importance weighting in Tighter Variational Bounds are Not Necessarily Better. The proof will be given for tree-structured coupling.

The second piece is the model. We will reimplement the Social-CVAE backbone: graph attention message passing over agents and a recurrent trajectory decoder. Its decoder will be modified so that agent $i$'s future depends on its own code and the codes of agents it attends to. The sparse attention mask then defines the factor graph.

When the mask contains loops of high treewidth, we restrict the coupling to a maximum spanning tree of the attention scores. We report how much attention mass this discards, because it is a real modeling restriction. This coupling is what makes the model a joint forecaster rather than a set of marginal forecasters.

Baselines

The naive reference point is the standard continuous trajectory VAE trained with the single-sample objective. On INTERACTION highway merging, Social-CVAE reports this model attends to approximately 8.68% of surrounding agents, measured by Agent Ratio. It reports minADE 1.86 and minFDE 4.17 on Argoverse, and FDE 3.85 on the INTERACTION generalization track (Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling).

The strongest existing method on our exact question is Social-CVAE itself, the conditional VAE with auxiliary decoder and sparse entmax attention. It targets the same failure, on the same benchmarks, with continuous latents. Its reported numbers are Agent Ratio 15.5% on highway merging, FDE 2.70 on the generalization track, and minADE 1.59 / minFDE 3.52 on Argoverse. We will rerun it from the authors' description with our training pipeline, so that both methods share data splits, backbone width, optimizer, and a hyperparameter search of equal size. Beating it means lower minADE/minFDE on Argoverse and the INTERACTION generalization track together with a higher Agent Ratio, without the auxiliary decoder. A tie on accuracy with a higher Agent Ratio would still be informative. Any gain on ETH/UCY matters most, because Social-CVAE's authors report their fixes were less effective there and give no number.

The discrete-latent alternative is Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data. It reports up to 60% ADE improvement over earlier generative methods on ETH/UCY, but no reported absolute result was found on Lacuna for its ETH/UCY ADE/FDE or nuScenes NLL. We will therefore run it ourselves on ETH/UCY and nuScenes with its released configuration.

Within the variational family we compare against four gradient-estimation variants, all with the same backbone and per-step FLOP budget. The first two are a joint $K$-sample objective with ordinary gradients and the same objective with DReG. The third is PIWAE from Rainforth et al., which trains the decoder on a tight objective and the encoder on a loose one. The fourth is the joint objective with the "sticking the landing" estimator, which drops the score term. Budgets are matched in decoder FLOPs, not in $K$: pairwise factors cost $K^2$ evaluations per edge, so joint baselines get correspondingly larger $K$.

Two ablations isolate the ingredients. One removes all coupling factors; the estimator then reduces to a sum of per-agent importance-weighted objectives, an independent-agent model with a tight bound. The other keeps the combinatorial estimator but uses ordinary gradients instead of DReG.

Why not just…?

Why not just use discrete latents, as Trajectron++ does? Discrete codes cannot collapse in the Gaussian-KL sense, and Trajectron++ performs well. A small codebook, however, restricts each agent to a fixed set of intents. Interactive merges unfold as continuous negotiation over long horizons; Generic Probabilistic Interactive Situation Recognition and Prediction: From Virtual to Real models 9-second merges. This is a genuine competitor, not a strawman. If Trajectron++ matches our accuracy on nuScenes and ETH/UCY, the continuous-latent case rests only on the interaction-sensitivity probe described below, and we will say so.

Why not just keep Social-CVAE's auxiliary decoder and entmax attention? It works on vehicles, and we adopt its sparse attention as the source of our graph. Its authors report it is less effective on pedestrians. It also never changes the objective, which stays loose. If tight objectives add nothing once the auxiliary decoder is present, the "Social-CVAE plus our estimator" cell will show it, and the contribution shrinks to a statement about gradient estimation.

Why not just use joint importance weighting with DReG and a larger $K$? This is the obvious combination and the main threat. Our claim is that its useful $K$ grows exponentially with the number of agents, by the multiplicative-noise argument of Geffner et al. That argument was made for $\alpha$-divergences and must be checked here directly. The synthetic sweep below does so. If joint DReG's effective sample size does not decay with $N$ on real scenes, which have few interacting agents, our method's advantage will be small, and we will report that.

Why not just weight samples sequentially, agent by agent, with sequential Monte Carlo? Particle methods do reduce weight degeneracy. However, Auto-Encoding Sequential Monte Carlo reports that tighter particle objectives starve the proposal network of learning signal. Agents also have no natural order. We include an agent-ordered particle baseline in the synthetic study only.

Proposed Experiments

The first experiment is a synthetic scene with known ground truth. We use $N$ agents with linear-Gaussian latents coupled along a chain or tree, with $N$ from 2 to 32, so the exact posterior and likelihood are available. For each estimator we measure three quantities against $N$ at fixed compute: effective sample size, the gap between the objective and the true log-likelihood, and the encoder gradient signal-to-noise ratio. The decisive prediction is that joint weighting's effective sample size and bound improvement decay roughly exponentially in $N$, while ours decay at most linearly. Coupling strength is swept to map the intermediate regime.

The second experiment uses the real benchmarks: INTERACTION (highway merging and generalization tracks), Argoverse, ETH/UCY and nuScenes. We report minADE/minFDE, Agent Ratio, per-agent KL to the prior, and importance-sampled test log-likelihood with a large evaluation $K$ shared across methods.

The third is an interaction-sensitivity probe. We swap a neighbor's latent code or history and measure how much the target agent's forecast changes. This tests use of neighbor information directly rather than through attention weights.

We will also track encoder gradient signal-to-noise ratio during training for every method. The aim is to test the causal story: social collapse should coincide with low encoder signal in the joint baselines and not in ours.

The proposal fails if any of three things happens: the synthetic sweep shows no exponential degradation for joint DReG; the coupled estimator does not raise Agent Ratio or sensitivity over the uncoupled ablation; or Social-CVAE with its auxiliary decoder already saturates both.