A Growing Library of Two-Agent Interaction Modes for Joint Motion Forecasting

When two cars approach the same merge or crossing, what matters for a self-driving planner is less where each car will be and more how the encounter resolves: who goes first, by how much margin, whether one car hesitates and then commits. Today's joint forecasters answer this with a fixed number of learned output slots, usually six, shared across every scene. This has two costs. A simple car-following pair has one plausible outcome, so the extra slots fill with small speed variations of it. A dense four-way negotiation may have more distinct outcomes than there are slots. And when the system reaches a new city whose driving culture produces an encounter type it has never seen, it has no place to put it.

We propose to learn a library of interaction modes whose size is inferred from data rather than fixed. The clustering runs over pairwise relational features: who passes the conflict point first, the time gap at that point, and the relative speed profile. It uses a hierarchical Dirichlet process, a Bayesian prior that lets many groups of data (here, map locations) draw from one shared pool of clusters that can grow without a preset limit, while each group uses only a few of them. A forecaster then predicts, from past context, which library modes are live in the current scene, and decodes joint trajectories conditioned on them. The central bet is that three things are cheap in this representation and hard for fixed-slot forecasters: the per-scene mode count, cross-location sharing, and expansion into new encounter types.

Core Claim

Clustering joint futures in relational coordinates, under a shared-but-growing nonparametric prior, recovers distinct interaction outcomes rather than geometric variants of one outcome. It also lets the mode set expand when a forecaster trained in some locations is deployed in others. The claim is falsifiable in two separate places. First, the relational coordinates should beat the same clustering run on raw agent-centric trajectories. Second, the growing library should beat a fixed-size library whose size was tuned on validation data, and this advantage should show up specifically on held-out locations. If the tuned fixed library matches us under location shift, the nonparametric machinery is not earning its keep, and we will report that.

The closest existing work is DeLL. It grows a Dirichlet-process mixture of ego trajectory and perception anchors for lifelong end-to-end driving in CARLA, and reports substantial compute overhead from alternating deep training with mixture updates. We differ in three ways: we cluster reciprocal two-agent outcomes rather than the ego's own maneuvers, we use a hierarchical prior across locations, and we fit the library offline with a fast deterministic objective instead of alternating inside training. CVQ-VAE scenario clustering motivates the relational choice. It shows that clustering highD scenes on raw kinematics groups them by superficial features, and that injecting an interaction signal from a social-force heuristic cuts cluster entropy from 3.014 to 1.243. Its codebook size is fixed, however, and its interaction signal is hand-designed. GAME-UP shares our goal of covering distinct social outcomes rather than Euclidean spread. It gets there by solving for local Nash equilibria with estimated payoffs during training, which is expensive and assigns no prior frequency to modes. Our approach is probabilistic and replaces equilibrium solving with clustering of observed outcomes.

Research Program

Relational descriptor. For every interacting pair in the training data, we compute a fixed-length descriptor $r$ of the realized future. It contains three things: the sign of passing order at the predicted conflict region, as used in braid-style topology from BeTop; the post-encroachment time gap; and a low-dimensional principal-component summary of the relative longitudinal and lateral motion over the horizon. This descriptor ignores absolute speed and position, which is what should prevent the speed-variant over-splitting that LookOut calls the "diversity curse."

Library fitting. Full sampling-based inference for Dirichlet processes is too slow for datasets of this size. We therefore use the small-variance limit, in which the Bayesian posterior collapses to a hard-assignment objective, following the construction DP-space used to reach roughly 250× speedups on Hopkins 155. For the hierarchical case, the objective is

$$\min_{z,\mu,K}\; \sum_{s}\sum_{(i,j)\in s} \lVert r_{ij} - \mu_{z_{ij}} \rVert^2 + \lambda_{\ell} \sum_s K_s + \lambda_g K .$$

Here the terms mean the following. $s$ indexes map locations. $(i,j)$ ranges over interacting pairs at that location. $z_{ij}$ is the pair's library mode and $\mu_k$ is mode $k$'s center. $K_s$ is the number of modes location $s$ uses, and $K$ is the size of the global library. $\lambda_\ell$ and $\lambda_g$ are the costs of opening a mode locally and globally.

DP-space and the research file both flag that these penalties need calibration. We will set them by held-out predictive likelihood of the corresponding soft mixture and report full sensitivity curves, not a single chosen value. Because temporal "chattering" between modes was a known failure of sequence-level nonparametric models in sticky HDP-SLDS, we cluster whole-horizon outcomes rather than per-timestep states. This deliberately avoids the stickiness trade-off at the cost of not modeling mid-horizon renegotiation. We state that as a limitation.

Forecaster. On a MultiPath++ backbone MultiPath++, we add a head that outputs a distribution over library modes for each agent pair from past context only. Each mode center has a learned embedding used as a decoder query. The decoder produces the joint pair trajectory under that mode, and the top modes fill the benchmark's required prediction slots. If fewer modes are live than there are slots, the remaining slots go to within-mode samples. Mode assignment is a single forward pass, so inference cost matches the backbone.

Growing the library. To handle new locations, we refit the objective offline on newly logged encounters. The existing centers are frozen and only new modes are allowed, and only the new query embeddings plus the mode head are fine-tuned. At run time, a pair whose predicted descriptor falls outside every live mode by more than the $\lambda_g$-implied radius is flagged as a novel encounter. This flag is a deliverable in its own right, because a planner could fall back to conservative behavior when it fires.

Baselines

The naive reference point is the MultiPath++ backbone decoding each agent independently with its fixed learned anchors. JFP's comparison shows this reference yields a WOMD Interactive overlap rate of 0.064, corrected here: that number comes from JFP. JFP itself is the strongest structured joint decoder baseline. It reconciles per-agent proposals with pairwise collision energies through three rounds of belief propagation and reports an overlap rate of 0.030 on WOMD Interactive validation.

For mode coverage, the strongest competitor is NashFormer from GAME-UP. It reports a WOMD Interactive miss rate of 0.483 against 0.508 for its base model, and 33.6% more interactions covered. BeTopNet ranked first on the Waymo joint leaderboard at submission and has public code (BeTop codebase). The file gives no absolute joint mAP value for it, so we will run it ourselves rather than quote a number.

For fairness, all methods we train ourselves share the same encoder, the same training data, the same number of output slots, and the same GPU-hour and hyperparameter-search budget. Published numbers will be quoted only alongside our re-runs.

We will beat the strongest baseline if two conditions hold together. In distribution, we must lower miss rate below NashFormer's 0.483 on WOMD Interactive while keeping overlap at or near JFP's 0.030. Under location shift, we must show a clear drop in miss rate on held-out locations relative to all baselines retrained on the same source locations.

The key ablations isolate each ingredient. The first uses the same growing hierarchical library but fits it on agent-centric absolute trajectories instead of relational descriptors, which isolates the representation. The second uses a fixed-size k-means library on the relational descriptors, with $K$ tuned on validation data, which isolates the nonparametric prior. The third uses a flat Dirichlet-process library without location groups, which isolates cross-location sharing.

Why not just…?

Why not just use a fixed library with a tuned $K$? This is the real threat. In distribution, a well-tuned $K$ may match us, and the relational descriptor could be doing all the work. The fixed-$K$ relational ablation is designed to settle it. The prior's specific advantages are that it adapts the per-location mode count and adds modes under shift without re-choosing $K$. So the decisive comparison is on held-out INTERACTION locations, with the fixed library refit under the same new-data budget. If fixed-$K$ ties there, we will conclude that the contribution is the descriptor, not the prior.

Why not just enumerate braid topologies as discrete modes? Crossing order from BeTop is exactly one coordinate of our descriptor. It separates "A goes first" from "B goes first," but it cannot separate a smooth zipper merge from a late forced gap acceptance, because both have the same order. We will run a topology-only library as a baseline. If it matches us on miss rate, the continuous coordinates are unnecessary.

Why not just push geometric diversity with farthest-point sampling or non-maximum suppression? GAME-UP and LookOut both report that Euclidean spread over-samples variants of one social outcome and misses the alternative one. We will include a diversity-sampling variant of the backbone. We will measure outcome coverage directly, as the fraction of held-out pairs whose realized passing order appears among the predictions, not only displacement miss rate.

Why not just learn the mode set end-to-end, as MultiPath++'s learned anchors or a mixture density network do? Learned anchors cannot grow after training. On highway ramps, the fatality-aware benchmark found that a mixture density network had the best plain Brier score (0.1099) but high non-defensiveness (0.1303), which made its fatality-weighted Brier (0.2361) worse than inverse reinforcement learning (0.2053). That result warns that likelihood-trained mode sets can drop rare aggressive outcomes. We will report the same weighted metric where the benchmark is available.

Proposed Experiments

The primary in-distribution test is on the WOMD Interactive split WOMD. We report miss rate, joint mAP, minADE/minFDE, overlap rate, and outcome coverage against the baselines above.

The decisive experiment is on the INTERACTION dataset, which spans roundabouts, intersections, and ramps across 11 locations in four countries. We train on a subset of locations and test on locations held out by country. Lacuna reports no absolute prediction numbers for this dataset, so all baselines will be run by us. We measure how many new modes the library opens, whether those modes correspond to encounter types absent from training (checked by manual inspection of a random sample by two annotators), miss rate and outcome coverage before and after library growth, and how well the novelty flag predicts large-error pairs, scored by the area under its precision-recall curve.

As a representation check on highD, we apply the library to the scene extraction protocol of CVQ-VAE. We compare cluster entropy against their 1.243 and assignment accuracy on their ghost-vehicle augmented set against their 56.8%, testing whether relational descriptors match their domain-knowledge guidance without a hand-designed social-force term. On NGSIM ramp merges, we report the fatality-weighted Brier score against the HMM, MDN, and IRL values quoted above.

Finally, we report library-fitting wall-clock time and per-scene inference latency against DeLL's alternating scheme. This tests whether the offline small-variance fit removes the overhead that paper reports.