A forecaster that predicts where road users will move over the next few seconds usually learns habits of the city it was trained in: typical speeds, the side of the road people drive on, and where pedestrians tend to cross. Those habits break in a new city. What does carry over is physics. A bus cannot turn sharply and must stay on the road, while a pedestrian can stop and pivot on the spot. We propose to teach this physics to a small forecaster through an extra training target that is thrown away before deployment. For each agent, the model learns to predict how quickly an agent of that type, starting from its current position and speed, could physically reach every nearby map cell. We then test whether this target makes forecasts transfer across cities, for example from Boston to Singapore within nuScenes, better than the simpler fixes a reviewer would reach for first.
The target costs nothing at test time. That matters because the forecaster has to run on in-vehicle hardware. For this reason we build its map encoder from depthwise separable convolutions: layers that split an ordinary convolution into a cheap per-channel spatial filter followed by a 1×1 layer that mixes channels, which cuts cost to roughly $1/N + 1/K^2$ of a standard layer (Xception). This efficiency constraint also raises a real scientific question. Does a heavily factorized encoder have enough capacity to absorb the extra geometric supervision, or does the supervision recover what the factorization gives up?
Core Claim
The claim is that a reachability field that depends on agent class and agent state is a training-only target that carries across cities. It encodes map-constrained kinematic limits, which are shared across cities, and leaves out behavioral preferences, which are not. Supervising the shared map-and-state representation with this field should shrink the drop in accuracy between the training city and a new city. It should do so more than three alternatives: using a linear motion prior, applying generic auxiliary tasks, or augmenting the data.
Concretely, the target for agent $i$ of class $c$ is the minimum time $T_c(x \mid s_i, M)$ for that agent to reach map cell $x$. Here $s_i$ is the agent's current state (position, heading, speed) and $M$ is the map. The time is computed offline by fast-marching over a kinematic model for that class: a unicycle model for pedestrians and a bicycle model for vehicles and cyclists, as in Trajectron++. Each class gets conservative physical speed and turning limits that are identical in every city, and each class has its own set of allowed map regions. The model minimizes
$$\mathcal{L} = \mathcal{L}_{\text{forecast}} + \lambda \sum_{x \in \mathcal{W}_i} \left| \hat{T}_c(x) - \min(T_c(x), T_{\max}) \right|,$$
where $\mathcal{W}_i$ is a local window around the agent, $T_{\max}$ caps the horizon, and $\lambda$ weights the auxiliary loss. The head that produces $\hat{T}_c$ is deleted after training.
This differs from the closest work in specific ways. Huang et al.'s motion-prior method makes cross-domain forecasting more robust by predicting residuals on top of a constant-velocity prior that stays in the model at test time. That prior is linear, ignores the map, and is the same for every agent type, so it cannot express that a cyclist and a bus starting from the same state have different reachable sets. SSWDP also uses a discarded auxiliary head, but its target is synthetic Gaussian noise on the waypoints of homogeneous walkers. Its target is not geometric in the map sense, it is not class-specific, and it was never tested across cities. Trajectron++ enforces kinematics by integrating predicted controls at the output, which cut constraint violations by more than 75%. It never supervises the internal representation with map-aware reachability, and it was evaluated only on standard nuScenes splits. MGMapNet uses auxiliary geometric losses, but only for static road elements. Our target is dynamic because it depends on each agent's state, which addresses the gap in the literature between static and dynamic geometry.
Research Program
Three prior results motivate the approach. Huang et al.'s geometric saliency work found that a CNN's F-score fell from 0.74 to 0.10 across cities, while an invariant corner-and-junction prior held 0.59. That is evidence that geometric structure transfers where learned appearance does not. However, their prior was handcrafted and fixed at test time, whereas ours is a learned internal target. Saha et al. found that training data spread across diverse regions beats data from nearby regions (Kappa 0.32 against 0.12). This constrains our design: any gain we report must survive a baseline that trains on more diverse data. Social LSTM and TraPHic show that agent classes differ in how they occupy space. This is the reason the target must be class-specific rather than shared by all agents.
The system has three parts. The backbone reuses the Trajectron++ graph encoder and its kinematic output layer. The rasterized map encoder is rebuilt in two matched versions: one with standard convolutions and one with depthwise separable convolutions. Following Chollet's finding that nonlinearities between the depthwise and pointwise steps hurt performance, the separable blocks keep that step linear. The auxiliary head then attaches only after full blocks, so it never adds an activation inside one. The reachability head is a small decoder that reads the agent's state embedding together with local map features and outputs $\hat{T}_c$ on a 64×64 window. The fast-marching labels are precomputed once per agent-timestep.
The analysis has two parts. First, we measure the generalization gap, meaning error in the new city minus error in the training city, broken down by agent class. The prediction is that the largest gains come for cyclists and pedestrians. For these agents a constant-velocity prior is weakest, and the tension between rigid priors and nonlinear motion by vulnerable road users is most acute. Second, we cross encoder type (dense or separable) with the auxiliary loss (on or off). This asks whether the auxiliary supervision helps the smaller encoder more, which would mean it compensates for the lost capacity, or less, which would mean the factorized encoder is a bottleneck. Either answer is informative. The second would be a useful negative result for anyone designing efficient multi-task models.
Baselines
The naive reference point is Trajectron++ trained only on Boston nuScenes scenes and tested zero-shot on Singapore scenes, with no changes. Next to it we report pure constant-velocity extrapolation. This tells us whether any learned model beats physics-free geometry once the city changes.
The strongest existing competitor is the constant-velocity residual method of Huang et al.. The reported numbers are from the file. When Trajectron++ transfers from nuScenes to Lyft, their method reduces ADE1 (average displacement error of the single most likely trajectory) from 85.19 to 25.72. On their 20 pedestrian cross-domain tasks it reaches an average ADE of 0.44 and FDE (final displacement error) of 0.72, compared with 0.52 and 1.04 for the test-time adaptation baseline TTA-GNN. We found no reported result on Lacuna for Boston-to-Singapore transfer within nuScenes, so we will reproduce this baseline on that split ourselves. We will also add their prior on top of our model. The goal is to test whether the reachability target adds something beyond the linear prior, not merely whether it replaces it. Beating this baseline means lower minADE and minFDE, the best-of-K versions of the two error measures, on the new city, averaged over all classes. The improvement must hold across at least three seeds and must not come with worse error in the training city.
The auxiliary-task competitor is SSWDP-style waypoint-distortion prediction applied to the same heterogeneous backbone. It is the closest method that also relies on a training-only head. The augmentation competitor is Trajectron++ trained with mirror flips and rotations. Mirror flips are chosen deliberately because they directly simulate the change between right-hand and left-hand traffic. The diversity competitor trains on a mix of scenes from two nuScenes locations and tests on a third, held-out location.
Three ablations isolate the key ingredients. One uses a class-agnostic reachability target, identical for all agent types, to test whether heterogeneity matters. One uses a static signed-distance field of the drivable area, which does not depend on agent state, to test whether dynamics matter. One feeds the true reachability field as an input at test time instead of using it as a target, to test whether the benefit comes from shaping the representation or simply from the information the field contains.
For fairness, every method uses the same data, the same backbone, and the same number of training epochs. Every method also gets the same budget of 20 trials for tuning its own key hyperparameter, such as $\lambda$ or the noise scale. Model selection uses only a held-out validation set from the training city, so no method sees target-city data before testing.
Why not just…?
Why not just use Huang et al.'s constant-velocity residual prior? This is the most serious threat, since it already delivers large zero-shot gains. Its known weakness is that the prior is linear and blind to the map. That weakness should matter most for turning vehicles and for pedestrians who stop suddenly, but the file contains no per-class breakdown that confirms it. The combined model settles the question. If adding our target on top of the prior gives no further gain, especially for cyclists and pedestrians, then the prior already captures what transfers and our idea adds nothing.
Why not just add the SSWDP distortion task, which already works? SSWDP gives a 16.6% relative improvement in ADE on ETH/UCY and large gains under corrupted inputs (SSWDP). However, it teaches the model to handle local noise, not to respect map-constrained, class-specific limits, and it was never tested across cities. If its generalization gap matches ours, then any structured auxiliary task is enough, and the reachability content does not matter. We would report that outcome as it is.
Why not just mirror-augment or train on more diverse cities? Mirror flips cheaply address the change in driving side, and Saha et al.'s result suggests diversity is strong. If flipping alone closes the gap between Boston and Singapore, our gain may amount to a fix for driving side. We therefore also test on nuScenes-to-Lyft, where both datasets drive on the right, so flipping has little to fix there. We also report our method combined with augmentation to show whether the two gains add up.
Why not just use standard convolutions or a tiny graph network, since Social-STGCNN already has 7.6K parameters? The graph part of the model is already cheap. The cost lies in the rasterized map encoder that Trajectron++ uses. We will measure wall-clock latency on embedded hardware, not just FLOPs, because depthwise operations are limited by memory bandwidth (Optimizing Depthwise Separable Convolution Operations on GPUs). If the separable encoder gives no real latency gain, it stays in the paper only as the capacity-reduced arm of the encoder-by-loss experiment.
Proposed Experiments
The primary benchmark is nuScenes split by location (nuScenes). We train on Boston Seaport and test on Singapore, and we also run the reverse direction. This transfer combines a change in driving side, different mixes of agent classes, and different road layouts. The second benchmark is nuScenes to Lyft, which follows the protocol of Huang et al. and so allows a direct numerical comparison with the 25.72 ADE1 figure. The third is ETH/UCY leave-one-scene-out, where the pedestrian-only setting lets us compare with the reported 0.44/0.72. Here the class-specific part of our method cannot help, so any gain must come from state-dependent geometry. We also report transfer from NGSIM to TRAF as a test of highway-to-dense-Asian-traffic heterogeneity. No reported result found on Lacuna for cross-dataset TRAF transfer, and TraPHic reports only a 30% RMSE improvement within each dataset, without absolute figures. Metrics are minADE and minFDE at K=1 and K=5, each broken down by class, the rate of map-constraint violations, the generalization gap, and parameters, FLOPs, and measured latency.
Three checks are decisive. First, the class-specific target must beat both the class-agnostic and the static ablations on the new city. If it does not, the core claim fails. Second, the reachability head, evaluated before it is discarded, must show low error in the new city even when the forecast degrades there. We will correlate the head's per-agent error in the new city with forecast error. A positive correlation indicates that the representation carries the transferable geometry, while no correlation suggests the gain comes from general regularization. Third, the gain must hold on nuScenes-to-Lyft, where mirror augmentation has little to fix, and it must hold for the separable encoder, not only the dense one.
References
- Xception: Deep Learning with Depthwise Separable Convolutions
- Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data
- Towards Single Source Domain Generalisation in Trajectory Prediction: A Motion Prior based Approach
- Enhancing Trajectory Prediction through Self-Supervised Waypoint Distortion Prediction
- MGMapNet: Multi-Granularity Representation Learning for End-to-End Vectorized HD Map Construction
- Accurate Building Detection in VHR Remote Sensing Images Using Geometric Saliency
- Trusting small training dataset for supervised change detection
- Social LSTM: Human Trajectory Prediction in Crowded Spaces
- TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions
- Optimizing Depthwise Separable Convolution Operations on GPUs
- nuScenes: A multimodal dataset for autonomous driving