When several cars approach a roundabout, the question that matters most for safety is often discrete: who goes first. A good forecast of the scene should put the right probability on "the truck yields, then the sedan merges" against the reverse order. The standard ways of training and scoring multi-agent trajectory forecasters measure something else. They use per-point coordinate errors, or a Wasserstein distance (the minimum cost of moving one probability distribution onto another) computed in raw coordinate space. Two joint futures can be a meter apart in coordinates and still disagree completely on who yielded. We propose to build, for each scene, a tree whose upper branches are these discrete interaction outcomes, such as the passing order at each conflict point, and whose lower branches refine the geometry. Forecasts are then trained and scored with the distance this tree defines.
The useful fact comes from tree-based optimal transport. On a tree, the transport distance has an exact closed form: a weighted sum, over edges, of how much probability each side puts below that edge. We show below that the matching scoring rule (a recipe for scoring a forecast against the single future that actually happened) has an equally simple form, a weighted Brier score over the tree's subtrees. It is proper, meaning it is minimized in expectation by forecasting the true distribution. It runs in linear time and needs no critic network and no Sinkhorn iterations. The claim to test is that training joint forecasters with this score gives better-calibrated interaction-outcome probabilities, and better downstream decisions, than coordinate losses or flat outcome classifiers, without hurting standard displacement metrics.
Core Claim
Fix a scene and let $d_{\mathcal{T}}$ be the path-length metric of a tree $\mathcal{T}$ whose leaves are cells of joint-future space. The closed form from Tree-Sliced Variants of Wasserstein Distances gives $W_1(\alpha,\beta)=\sum_e w_e|\alpha(T_e)-\beta(T_e)|$. Here $w_e$ is the length of edge $e$, $T_e$ is the subtree below $e$, and $\alpha(T_e)$ is the probability distribution $\alpha$ assigns to that subtree.
A trajectory dataset gives only one realized future $y$ per scene, not a target distribution. Plugging the Dirac mass $\delta_y$ into $W_1$ is a trap. The expected per-edge term is linear in $\alpha(T_e)$, so the loss rewards collapsing all probability onto the most likely outcome. We therefore use the energy score $\mathbb{E}\,d(X,y)-\tfrac12\mathbb{E}\,d(X,X')$, where $X$ and $X'$ are independent draws from the forecast. This score is proper whenever $d$ is of negative type. Tree metrics have that property; it is the same property that lets Le et al. build valid kernels from TSW. On a tree the expectations collapse, edge by edge, to
$$S_{\mathcal{T}}(\alpha,y)=\sum_{e\in E} w_e\big(\alpha(T_e)-\mathbf{1}[y\in T_e]\big)^2,$$
which is a hierarchical Brier score. Each edge charges the squared error in the predicted probability of one interaction event, weighted by how high that event sits in the tree.
Inside a single leaf, the tree metric treats all futures as identical. Scalable Nearest Neighbor Search for Optimal Transport found that costs read directly off tree edges distort geometry, and fixed it in Flowtree by using the tree only to route mass while evaluating cost with the original Euclidean metric. Following that lesson, we use the summed metric $d=d_{\mathcal{T}}+\lambda\|\cdot\|$. This sum is still of negative type, so the score stays proper. It is the hierarchical Brier term plus $\lambda$ times the ordinary Euclidean energy score. As a result, outcome-level errors are charged on the tree and within-outcome geometry is charged in meters.
Research Program
Tree construction. For each scene we find conflicting agent pairs: pairs whose paths cross, or whose time-to-conflict falls under a threshold, following the interaction notion in the INTERACTION Dataset. For each pair, the outcome of a joint future is computed geometrically from the trajectories: $i$ passes the conflict point first, $j$ passes first, or no conflict occurs. Pairs are ordered along the tree's depth, and edge weights decay with depth. Below the outcome levels sit maneuver classes, then a coarse spatial partition of each agent's endpoint.
The tree is a fixed function of trajectories within a scene, so it is a valid metric over all joint futures for that scene. Only leaves that receive probability need to be built, so the cost is linear in (number of samples) × (depth).
Handling the pair ordering. The order in which pairs appear along the depth is arbitrary, and this is where the "which trees to sample" problem raised by Le et al. shows up here. We average $S_{\mathcal{T}}$ over random orderings, which makes the loss a tree-sliced score. We also compare against a single ordering chosen by ego relevance.
Training. Mixture-style joint predictors output $K$ joint modes with probabilities $\pi_k$. Each mode's subtree membership is softened with a sigmoid on the arrival-time gap at each conflict point. That softening lets gradients reach both the probabilities and the geometry.
Analysis. We will characterize, on synthetic two-agent crossings, when the hierarchical score and the Euclidean energy score disagree. The expected regime is futures that are close in coordinates but differ in passing order. This is the "diversity curse" that LookOut handled with a handcrafted planning-diversity energy.
How this differs from the closest work. Two lines of prior work are nearest.
The first already uses optimal transport inside trajectory models. Social-WaGDAT matches latent distributions with a Wasserstein auto-encoder, whose Wasserstein term is computed by an adversarial critic in an unstructured latent space. We use no critic, and our tree encodes interaction semantics.
The second connects tree-structured graphs with multi-marginal transport: Multi-Marginal Optimal Transport and Probabilistic Graphical Models, Inference With Aggregate Data in Probabilistic Graphical Models, and On the complexity of the optimal transport problem with graph-structured cost. These solve entropically regularized transport over discrete, aggregate population histograms on a known tree. In their setting the tree is over variables. In ours it is over outcomes of a single scene, and we need no iteration.
FJMP motivates the construction. It learns directed acyclic graphs of influencers and reactors, which shows that traffic interactions factor into a few ordered relations. FJMP still trains with standard regression and classification losses. We reuse its relational structure as a source of conflict pairs and change only the objective.
Baselines
Naive reference points. The first is an independent per-agent predictor whose joint modes are formed by combining marginals. This is the setup that Generic Probabilistic Interactive Situation Recognition and Prediction identifies as producing inconsistent, colliding joint futures. The second is the same architecture as our model trained with its original loss. Comparing against the second shows what the objective alone contributes.
Strongest existing method on INTERACTION. This is FJMP, which ranks first on the INTERACTION v1.2 multi-agent test leaderboard. The file gives the ranking but not its error values, so there is no reported result found on Lacuna for FJMP's numerical joint metrics. We will retrain FJMP from its released configuration, keeping its architecture, data split, epochs and tuning budget. The only change is the added term $S_{\mathcal{T}}$, plus a matched sweep over $\lambda$ that gives the baseline loss's weights the same number of trials.
Generative baseline on Argoverse 2. This is the diffusion model OGD from Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation. It ranked 4th on the multi-world leaderboard using 40 denoising steps instead of 500. We compare against it in two ways. One is as an evaluated competitor at equal sample count. The other is as a teacher: a one-shot predictor is distilled from OGD's samples using tree-$W_1$, which is the case where both sides really are distributions.
Beating the baselines. Success means a statistically significant improvement in passing-order calibration error and in the decision test below, with joint minADE and minFDE no worse than the matched baseline within seed variance.
Ablations isolating the tree. Three ablations each remove one ingredient. (i) The Euclidean energy score alone ($\mathcal{T}$ removed) tests whether the tree adds anything beyond a proper distributional loss. (ii) A one-level tree, which amounts to flat cross-entropy-like Brier on the joint outcome label, tests whether the hierarchy matters. (iii) Tree-$W_1$ to the Dirac (improper) tests whether propriety matters; we expect overconfident outcome probabilities there.
Why not just…?
Why not just add a cross-entropy head that classifies yield or pass for each conflict pair? This is the most serious threat, and ablation (ii) together with a per-pair classifier baseline tests it directly. A per-pair head scores pairs independently. It cannot penalize joint outcome combinations that are each plausible but contradict one another. A flat joint classifier has exponentially many classes and shares nothing between related outcomes. If either matches our calibration and decision results, we will report that the tree adds nothing beyond outcome supervision and narrow the contribution to the evaluation metric.
Why not just run Sinkhorn or exact optimal transport with a cost that adds a penalty for differing passing order? With $K\approx 6$ modes, compute is not the obstacle; at this size the speedups reported by Le et al. (under 3 hours against over 8 days on CLASSIC) do not matter. The obstacle is that there is only one ground-truth future per scene. Transport to a single point reduces to an expected cost, which is improper. A hand-built penalty cost is not guaranteed to be of negative type, so its energy score need not be proper either. The tree gives propriety by construction. We will still include a Sinkhorn-to-Dirac variant and an energy score with the penalty cost, so this argument gets tested rather than assumed.
Why not just train with the planner's cost, as LookOut does? LookOut's diversity energy needs a specific planner inside the training loop and is tuned to that planner. Our score is planner-agnostic. We test whether it nonetheless gives a planner better inputs using the decision check below. If a LookOut-style energy wins clearly on that check, the tree score's value is limited to planner-free settings.
Why not just use one-dimensional sliced Wasserstein? Projecting joint futures onto random lines mixes all pairs' outcomes into a single coordinate. Le et al. found that tree slices beat line slices where structure branches. We include sliced Wasserstein as an energy-score variant, though we expect it to behave like the Euclidean ablation.
Proposed Experiments
We will train on INTERACTION v1.2 using the multi-agent track and on Argoverse 2 using the multi-world track. MultiPath++ (WOMD mAP 0.409, minADE 0.556 m, from MultiPath++) is a marginal benchmark, so it serves only as context.
The metrics are joint minADE and minFDE, cross-agent collision rate among predicted modes, the held-out hierarchical score, and expected calibration error of passing-order probabilities. To avoid grading the model on its own training target, calibration is computed with outcome labels from a second, independent definition: occupancy-overlap ordering rather than arrival time at the conflict point.
The decisive check is a decision test on logged INTERACTION conflicts. One agent acts as ego and picks among proceed, slow, and yield using the predicted joint futures. We measure how often its choice agrees with the choice an oracle makes using the true future. Fewer collision-implied and freezing decisions at equal displacement error would support the claim. Better calibration with no change in decisions would mean the tree improves bookkeeping but not behavior.
A final check on Argoverse 2 is whether a distilled student trained with tree-$W_1$ matches OGD's outcome distribution more closely than a student distilled with Euclidean energy distance.
References
- Tree-Sliced Variants of Wasserstein Distances
- Scalable Nearest Neighbor Search for Optimal Transport
- INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps
- LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving
- Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
- Multi-Marginal Optimal Transport and Probabilistic Graphical Models
- Inference With Aggregate Data in Probabilistic Graphical Models: An Optimal Transport Approach
- On the complexity of the optimal transport problem with graph-structured cost
- FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs
- Generic Probabilistic Interactive Situation Recognition and Prediction: From Virtual to Real
- Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation
- MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction