Reaction Lookup: Action-Conditional Motion Prediction by Billion-Scale Retrieval of Pairwise Interactions

A self-driving planner considers many candidate maneuvers every tenth of a second. For each one it needs to know how the cars and pedestrians around it would respond: whether the merging car yields if we speed up, or whether the pedestrian steps back if we creep forward. Neural predictors handle this question badly. Most predict each agent's future while ignoring what the ego vehicle plans to do. The few that condition on the ego plan learn the effect of that plan from logs in which the ego vehicle did only one thing. Large driving logs do contain the answer in a scattered form. Across the Waymo Open Motion Dataset's 100,000 twenty-second scenes (WOMD), almost every local two-agent situation (a car closing on another at a given gap, speed, and lane geometry) has occurred many times with different "actor" behaviors and recorded "responder" reactions. We propose to cut every logged scene into short pairwise interaction snippets and index them by the relative state of the two agents plus the actor's next three seconds. A query of the form "a responder in this situation, facing an actor who does this" then returns real recorded reactions.

The catch is scale, and that is where GPU similarity search comes in. Pairwise snippets multiply the corpus by the number of agent pairs and time steps, which gives roughly $10^8$ to $10^9$ keys. The planner issues thousands of queries per cycle, one per pair of nearby agent and candidate ego plan. Billion-key indexes built and searched on GPUs, using graph-based approximate nearest neighbor search (an index in which each stored vector links to a few near neighbors, so a query walks the graph instead of scanning everything), make this lookup feasible within a driving latency budget. The research question is whether action-conditional reaction prediction gets measurably better as the interaction memory grows toward that scale. If it does, the retrieval system is worth building. If accuracy saturates at a million snippets, the GPU-indexing half of the proposal is unnecessary, and we will report that.

Core Claim

Our claim is that a nonparametric "reaction memory" gives better-calibrated and more plan-sensitive predictions of other agents' responses to candidate ego plans than a parametric conditional predictor trained on the same logs. It does so because of two design choices. The first is pairwise, relative-frame decomposition, which turns one scene into many reusable interaction examples. The second is a key that encodes the actor's planned future, so lookups condition on the hypothetical action rather than on scene appearance. We further claim the gain grows with corpus size well past the point where exact brute-force search fits on one GPU. That is the condition under which GPU graph indexing is needed rather than merely convenient.

How this differs from the closest work

Three papers already retrieve driving scenes. DriveVLA-M0 keeps a memory of the planner's own past failures, retrieves the closest one by scene-structure similarity, and runs a quick low-rank weight update when similarity crosses a threshold. It reports 94.1 PDMS on NAVSIMv1 and a 15.19 ms query. Mütsch et al. learn self-supervised embeddings of whole traffic-scenario graphs on nuPlan for offline mining. Their embeddings find augmented copies of a scene 99.99% of the time, but up to 71% of data falls into an uninformative "sea of mundane driving." Spatial Retrieval Augmented Autonomous Driving keys retrieval on GPS. It helps static map estimation by +9.5 to +13.4 mAP but dynamic detection by only +0.04 to +0.10 NDS.

These results motivate our design. Whole-scene keys and location keys do not capture agent dynamics, and none of these systems condition on a hypothetical ego action. Our unit of retrieval is the pairwise snippet rather than the scene. The key includes the actor's planned trajectory. The retrieved payload is a set of recorded reactions, not a correction signal or a set of image features. ReMAP supplies a method we reuse. It pairs a frozen motion model with nearest-neighbor retrieval of motion templates and a small learned mapping, and reaches $R^2$ of 0.86–0.87 versus 0.74 for fine-tuning in a low-data regime. ReMAP is single-agent and retrieves one deterministic template, while we retrieve multi-modal reaction sets for interacting agents. Distributed Trajectory Similarity Search indexes isolated trajectories on CPU clusters for database queries. It neither models interaction nor feeds a predictor.

The action-conditional formulation comes from two sources. Prediction by Anticipation first moves the ego vehicle along a candidate action using a rule-based motion model, then learns how the occupancy grid reacts. LookOut argues that future predictions should be diverse in ways that change the ego's decision, not merely in endpoint distance. Both learn the reaction parametrically. We test whether looking up reactions does better.

Research Program

Snippet construction. For every ordered pair (actor $a$, responder $r$) within 40 m in WOMD training scenes, sampled at 1 Hz, we express both agents' last second of motion in the responder's frame. We add a compact summary of the responder's local lanes and the actor's next 3 s. The stored value is the responder's next 3–8 s. Pairs are filtered by a simple interaction test: predicted paths come within a threshold distance, or time-to-collision is below a bound. This keeps the memory from being swamped by the mundane driving Mütsch et al. observed.

Key learning. GPU indexes are built for plain Euclidean or inner-product distance, while what matters here is asymmetric and risk-dependent: a near-miss and a comfortable pass may look geometrically similar. We do not try to make the index support an exotic metric. Instead we train a small encoder $f$ so that Euclidean key distance predicts reaction difference:

$$\min_f \; \mathbb{E}\big[\,(\|f(x_i)-f(x_j)\|_2 - w_{ij}\, d(y_i, y_j))^2\,\big],$$

Here $x$ is a snippet's conditioning input (relative history, lanes, actor plan), $y$ is the responder's recorded future, $d$ is average displacement between futures, and $w_{ij}$ is a weight that increases with inverse time-to-collision. The weight follows the fatality-aware benchmark, so reactions near collisions are resolved finely in key space. This treats the metric-mismatch problem by learning it into the embedding, and leaves the index untouched.

Index. Keys are 64-dimensional. We build a graph index over $10^8$–$10^9$ keys with Tagore, which reports a 4.65× average construction speedup over CAGRA and cut Deep-1B construction from 14 hours to under 3 on one GPU. When the corpus exceeds device memory, we shard it with balanced graph partitioning and k-means tree routing from Gottesbüren et al., who report 1.19× to 2.14× throughput gains at 90% recall over balanced k-means sharding. A rough estimate shows why brute force fails at the top of this range. A billion 64-dimensional half-precision keys occupy about 128 GB, more than one GPU holds. Scanning them for 6,400 queries (64 agents, as in ProSim, times 100 ego candidates) costs about $4\times10^{14}$ multiply-adds per cycle. The memory is static and rebuilt offline. Streaming insertion is out of scope.

Prediction from retrieved reactions. We use two variants. The purely nonparametric one returns a mixture over the $k$ retrieved responder futures, each weighted by its key distance. The hybrid one feeds the retrieved futures as extra tokens into a lightweight decoder attached to a frozen MTR-style backbone, in the manner of ReMAP's frozen-model-plus-mapping design. When a responder has several relevant actors, the hybrid decoder attends over the retrieved sets for each actor. This pairwise factorization sidesteps the exponential joint space but can miss three-way effects. We will measure that failure mode explicitly.

Baselines

The naive reference point is a non-reactive predictor: the same frozen MTR-style backbone predicting each responder's future with no access to the ego plan. Any gain from conditioning must beat this. For orientation, on unconditional WOMD prediction the file reports MTR-A at 0.4594 Soft mAP and 0.1160 miss rate (MTR-A) and MultiPath++ at 0.409 mAP and 0.556 m minADE (MultiPath++). These are unconditional numbers. No reported result was found on Lacuna for ego-conditioned reaction prediction on WOMD, so we will report that baseline ourselves.

The strongest competitor is a parametric conditional predictor: the same backbone given the actor's future as an extra input token and fine-tuned on exactly the snippets in our memory. It sees the same information and data, and differs only in storing reactions in weights rather than in an index. It gets the same GPU-hours as our encoder plus decoder training, and the same hyperparameter-search budget. On NGSIM we also compare against inverse reinforcement learning, the strongest method in the fatality-aware benchmark: fatality-aware score $B_c$ = 0.2053, non-defensiveness 0.0698, conservatism 0.0178 (fatality-aware benchmark). Beating the parametric competitor means lower conditional minADE and negative log-likelihood on the interactive split, with confidence intervals from scene-level bootstrap, plus a higher plan-sensitivity score (defined below). On NGSIM it means lower $B_c$ than 0.2053 without raising conservatism.

Three ablations isolate the key ingredients. The first removes the actor plan from the key, which tests whether action conditioning matters. The second retrieves whole-scene embeddings in the style of Mütsch et al., which tests whether pairwise decomposition matters. The third sets $w_{ij}=1$, which tests the risk weighting. The system ablation replaces the graph index with exact flat GPU search at every corpus size where it fits, so recall loss from approximate search is measured rather than assumed.

Why not just…?

Why not just train a bigger conditional neural predictor on the same logs? This is the real threat. A model given the actor's future may interpolate reactions as well as a lookup. The only evidence the file offers is indirect: ReMAP's retrieval beat fine-tuning by over 0.1 $R^2$ in low data, and DriveVLA-M0 used retrieval to fix rare failures. Neither settles the interactive case. Our head-to-head comparison at matched data and compute settles it. We expect retrieval to help most in the rare, high-risk tail of snippets with low time-to-collision, and we will report results by risk bin so a win confined to easy cases is visible. If the parametric model ties everywhere, the proposal reduces to the hybrid decoder, and we would say so.

Why not just use exact brute-force GPU search over a smaller memory? If accuracy stops improving at a few million snippets, flat search is sub-millisecond and the billion-scale index is unnecessary. The corpus-scaling experiment answers this directly. A positive result requires that accuracy, especially in the high-risk bins, still improves between $10^7$ and $10^9$ keys, and that the graph index keeps recall above 95% of exact search within the latency budget.

Why not just retrieve whole similar scenes, as DriveVLA-M0 and scenario-graph search do? Whole-scene keys are dominated by layout and agent count, and they do not encode a hypothetical ego action. The near-zero dynamic-agent gains from location-based retrieval, and the 71% unclustered mundane mass found by Mütsch et al., suggest scene-level similarity is a poor proxy for reaction similarity. The whole-scene ablation measures this directly.

Proposed Experiments and Decisive Checks

The main experiment uses the WOMD interactive split, which labels pairs of interacting agents. We condition on one agent's ground-truth future and predict the other's, reporting conditional minADE, minFDE, miss rate, and negative log-likelihood. We also train the nonparametric and hybrid variants with corpora of $10^6$, $10^7$, $10^8$, and the full set of keys, and plot accuracy against corpus size by time-to-collision bin.

The counterfactual check targets what factual metrics cannot see: whether predictions change correctly when the ego plan changes. On held-out scenes we mine, with exact search, "natural experiment" pairs, meaning two real snippets with nearly identical responder history and context but different actor futures. Plan sensitivity is scored as the correlation between the predicted change in responder behavior and the recorded change. A predictor that ignores the plan scores zero.

The safety check transfers the memory construction to NGSIM highway merges and evaluates conservatism, non-defensiveness, and $B_c$ against the reported inverse reinforcement learning numbers.

The systems check measures end-to-end latency for 6,400 queries on one data-center GPU and on a sharded multi-GPU index. It reports recall against exact search, and verifies the whole query fits well inside DriveVLA-M0's 15.19 ms retrieval figure, which serves as a practical budget reference.

The proposal succeeds only if three things hold together: retrieval beats the matched parametric predictor on high-risk snippets, the gain still grows past the size where flat search fits on one GPU, and plan sensitivity is clearly above the non-reactive baseline. If any one of these fails, we report which part of the idea it rules out.