Where Drivers Behave: A Location-Keyed Memory for Continual Trajectory Prediction

Drivers behave differently in different places. A given on-ramp may have aggressive mergers. A given intersection may have a habitual rolling stop. A given bus stop may create a moving bottleneck every few minutes. Learned trajectory predictors, the models that forecast where surrounding vehicles will go over the next few seconds, have two ways to handle this. They can absorb these local habits into their shared weights, or they can re-infer them from what the camera sees right now. Both are fragile when a fleet keeps encountering new locations. Fine-tuning on a new city overwrites what the model knew about the old one. Re-inferring from perception fails when local cues are weak.

We propose to store local driving behavior where it belongs: in a memory indexed by map location. Each lane segment or grid cell holds a small running summary of how vehicles have actually moved through it. A frozen prediction network reads these summaries for the cells an agent is about to traverse. Learning a new city then means writing to new cells, not changing weights. Forgetting an old city would require overwriting its cells, which only happens if vehicles drive there again.

Core Claim

Our claim concerns a frozen trajectory predictor that reads from a coordinate-keyed memory written only by averaging, with no gradient steps. Trained on a stream of locations, it will match fine-tuning and replay on newly seen locations. It will show near-zero forgetting on old ones. It will improve steadily on locations it revisits, and that improvement should grow with the number of prior visits.

The memory's size grows with the area driven, not with the number of hours driven. It stores no raw trajectories.

The test that matters is whether the gain is truly spatial. If we scramble which cell's summary is attached to which location, the improvement should vanish. If cells with the same lane geometry but different observed behavior receive different summaries, accuracy should improve beyond what the map geometry alone provides. If either check fails, the memory is just extra capacity, and the idea is wrong.

How This Differs From What Exists

Continual-learning work on vehicle behavior keeps its memory in abstract feature space, not physical space.

  • NOMAD grows a bank of latent clusters, using a Dirichlet process mixture (a clustering model that adds new clusters when data no longer fits the old ones), and conditions diffusion experts on them. It rehearses old regimes with generative replay. Its authors report degradation where local semantic cues are weak, such as unmarked lots.
  • D-GSM allocates an episodic buffer of stored trajectories according to behavioral divergence between scenarios. It constrains gradients so that loss on those exemplars does not rise. Its scenarios are indexed by tag, not by place.

Neither can say "at this ramp, mergers accept short gaps." We motivate our work from their gap and compete against them directly.

The closest structural precedent is Expand and Compress (EAC). EAC freezes a spatio-temporal graph forecaster and gives each new sensor its own learned prompt vector. That is a form of location-specific parameters. We differ from EAC in three ways:

  • EAC's per-location vectors are learned by backpropagation at fixed sensor nodes. Ours are written by averaging at any map coordinate a vehicle passes through.
  • EAC forecasts aggregate flow at sensors. We forecast individual vehicle trajectories.
  • EAC never revisits and updates a prompt as conditions change.

Whether gradient-written location vectors beat average-written ones is an empirical question, so we include it as our key ablation rather than dismissing it. TFMoE takes the opposite stance, grouping sensors by behavior "over proximity." That choice is a useful foil: we predict it helps at the macroscopic level but not for micro-level interactions tied to particular road geometry.

On the spatial-memory side, our methods are borrowed:

  • MapNav shows that a top-down annotated map can replace frame buffers at about 1% of the memory.
  • SOMA keeps object memory persistent with exponential-moving-average updates that a policy attends to.
  • Memorize What Matters (3DGM) builds consensus over repeated traversals of the same streets. It explicitly discards moving vehicles as noise.

All three assume a static or quasi-static world and wipe or ignore dynamic content. We keep the persistent, coordinate-indexed ledger but fill it with exactly the dynamic content they discard.

Replay can provably increase forgetting constrains our design. It shows that replaying small subsets can turn vanishing forgetting into constant forgetting, non-monotonically in buffer size. That is one reason we avoid replay entirely. It also means replay baselines must be tuned over buffer size, not run at one setting.

Research Program

The memory is a sparse hash map from location keys to cell summaries.

Location keys. On datasets with HD maps, a key is a lane identifier plus a longitudinal bin of a few meters. Elsewhere, a key is a metric grid cell.

Cell contents. Each cell holds a visit count and $K$ prototype vectors. The prototypes summarize the interaction features of vehicles that passed through. Features come from a frozen encoder applied to each observed agent's short history and neighbors. We use several prototypes rather than a mean because driver behavior is multimodal and depends on context (Li et al.; Bench4Merge). A single average would blur free-flow and stop-and-go regimes together.

Write rule. When an observed feature $\phi$ lands in cell $c$, the nearest prototype $m_{c,k}$ in that cell moves toward it:

$$m_{c,k} \leftarrow (1-\alpha_c)\, m_{c,k} + \alpha_c\, \phi, \qquad \alpha_c = \max(1/n_c,\ \alpha_{\min}),$$

Here $n_c$ is the cell's visit count and $\alpha_{\min}$ is a floor on the update rate. The rule behaves like a running mean early on and like a decaying average with a fixed half-life once the cell is well visited. The floor exists to let stale behavior fade, for example after a lane closure, rather than persist forever.

Read and training. The predictor receives the scene and map as usual. It also receives the prototypes of the cells along each agent's candidate lanes, via cross-attention, together with $\log n_c$ so it can learn to discount sparsely visited cells. The predictor is trained once, on an initial set of locations. During that training we build memories with held-out traversals and randomly drop cells, so the network learns both to use memory and to fall back when it is empty. The predictor is then frozen for the rest of the stream.

Scope. The main variant has zero weight updates after deployment. A second variant adds one shared low-rank adapter, updated slowly, for cases where the frozen encoder lacks the features a new city needs. That plasticity question is the main honest risk of the design.

Analysis. We will measure how error at a cell falls with its visit count. We will also measure how quickly memory recovers after a deliberately injected behavior change, such as a synthetic lane closure in simulation. Recovery speed tests whether $\alpha_{\min}$ trades staleness against noise as intended.

Baselines

Do nothing. The naive reference point is the same predictor with its memory permanently empty. It is trained on the initial locations and run frozen on the stream. This is a map-conditioned model with no location history. It separates "the backbone is good" from "the memory helps."

Naive fine-tuning. The same backbone fine-tuned on each new location block in sequence. It gives the forgetting floor. D-GSM reports average displacement error rising by over 100% under vanilla sequential training on INTERACTION. We expect to reproduce a spike of that order, which serves as a sanity check on our protocol.

Strongest existing continual method on INTERACTION. D-GSM: divergence-weighted episodic memory with gradient projection. It is the strongest competitor here because it was designed for this dataset's sequential scenario protocol. We reimplement it on our backbone with the same total storage budget in bytes. Its buffer size is tuned on the first two tasks only, matching our tuning of $K$ and $\alpha_{\min}$. On the file's evidence, D-GSM reports the vanilla ADE spike above and a 40% to 50% reduction in training time and data use. No reported absolute ADE result for D-GSM was found on Lacuna. We will therefore report its numbers under our reimplementation. Beating it means equal or lower average minimum ADE across all tasks, together with lower forgetting (backward transfer closer to zero), at no more storage.

Experience replay and joint training. Experience replay is swept over buffer sizes, given the non-monotonic behavior predicted by Mahdaviyeh et al.. Joint retraining on all data serves as the upper bound we do not expect to beat.

Planning baseline on nuPlan. For the downstream planning check, the reference is NOMAD. It reports a 9.4% improvement in interPlan score over its baselines. We report our planner's interPlan score under the same closed-loop protocol, with the memory-augmented predictor feeding a fixed planner.

Key ablations. Two ablations isolate the proposal's ingredients:

  • Gradient-written memory (EAC-style). Each cell gets a learned vector trained by backpropagation on that cell's data, with the backbone frozen as in EAC. It is matched in parameter count to our prototypes.
  • Feature-keyed memory. The same prototypes are retrieved by nearest-neighbor similarity of the current scene's features rather than by location.

The first ablation tests whether averaging is enough. The second tests whether spatial keying, as opposed to content-based retrieval, carries the gain.

Why not just…?

Why not just condition on the HD map, since geometry already says where the merge is? Geometry tells the model that a merge exists. It does not tell the model how drivers there actually accept gaps. The do-nothing baseline is exactly this map-conditioned model. The geometry-matched check below (cells with near-identical lane topology but different observed behavior) directly measures whether memory adds information beyond the map. If memory adds nothing on geometry-matched pairs, this alternative wins and our claim fails.

Why not just learn a vector per location by gradient, as EAC does for sensors? This is a real threat. Gradient-written vectors may fit local behavior better than averages. Their costs are that they need labels, backward passes, and per-cell optimization at write time, and they cannot easily be decayed when behavior changes. The matched ablation settles which approach is better. If gradient-written cells win clearly on accuracy, we will adopt them and keep only the spatial keying and the decay rule as contributions. The paper would then become "location-keyed prompts for micro-level prediction with staleness control."

Why not just use replay or generative replay, which NOMAD and TFMoE already show work? They do work in their settings. TFMoE, for example, reaches a 60-minute mean absolute error of 17.20 on PEMSD3-Stream versus 21.22 for TrafficStream. But replay buffers store raw or synthesized trajectories, their cost grows with time, and their benefit can be non-monotonic in buffer size. The buffer-size sweep and storage-matched comparison will show whether replay reaches our accuracy at equal bytes. If a well-tuned replay matches us at equal storage, our advantage reduces to privacy and the revisit curve, and we will say so.

Why not just retrieve similar past scenes by feature similarity, without coordinates? Content-based retrieval avoids depending on localization. However, it conflates places that look alike but behave differently, which is precisely the case of interest. The feature-keyed ablation tests this directly.

Proposed Experiments

INTERACTION. We follow D-GSM's five-scenario sequential protocol across intersections, roundabouts, and ramps. We report minimum ADE and FDE (average and final displacement error) per task after each stage, average accuracy, backward transfer, and memory bytes.

nuPlan by city. Each of Boston, Pittsburgh, Las Vegas, and Singapore forms a stage (nuPlan). This tests cross-city transfer, which the file identifies as a known failure of fine-grained models.

nuPlan revisit stream. Within the cities, we order logs by date so the same road segments recur across traversals, as exploited in 3DGM. Here we plot per-cell error against prior visit count and measure forward transfer on revisits.

Decisive checks. Three outcomes would make the result convincing:

  • Key-scrambling. Randomly permuting cell summaries among locations, matched for visit count and lane type, should remove most of the gain over the do-nothing baseline.
  • Geometry-matched pairs. On cell pairs with matched topology, the memory variant should cut error relative to the map-only baseline.
  • Visit-count trend. The revisit curve should slope downward with visit count, while the feature-keyed and do-nothing baselines stay flat.

Staleness test. We inject behavior changes into a block of cells in the replayed stream, for example by substituting logs from a congested period. We then measure how many traversals the memory needs to recover, as a function of $\alpha_{\min}$.

Planning check. A secondary nuPlan closed-loop run reports whether better per-location prediction moves the interPlan score relative to NOMAD's reported improvement.