Many motion forecasters for self-driving cars do not regress future coordinates directly. They pick from a menu of a few thousand physically drivable candidate paths, which avoids averaging two plausible futures into one impossible one. In these classifiers each candidate is just an index, "class #184". When such a model is trained on cheap simulated traffic and then deployed on real roads, it learns how often the simulator's scripted drivers pick candidate #184 and how they execute it, and both are wrong in reality. We propose to give every candidate, in every scene, a short programmatically generated description of what that path does relative to the scene. An example is "turns left across the oncoming lane, enters the conflict zone after the oncoming car, stops before the crosswalk." The classifier then scores candidates by matching a scene encoding against an embedding of that description, not against a free per-index weight.
The bet is that the meaning of such a description is the same in simulation and in reality even when its frequency is not. Simulated drivers yield too often or too rarely, and they brake with idealized profiles, but "yields to the oncoming car" means the same thing in both worlds. If this holds, most of the simulator gap becomes a gap in label frequencies, which a handful of real examples can correct, plus a gap in fine-grained execution, which the fixed physically feasible menu already constrains. The proposal tests this decomposition directly. It also tests the uncomfortable possibility that plain structured attributes, with no language at all, do just as well.
What exists and what is missing
Three lines of work frame the opportunity. Discrete trajectory classification supplies the output space. CoverNet classifies over fixed menus built for ε-coverage of real data, or over dynamic menus generated from a bicycle model. On nuScenes it reports a 33% top-5 hit rate within 2 m against 13% for MultiPath and 10% for MTP. Trajeglish tokenizes motion into 384 displacement templates and transfers across cities (Waymo to nuScenes). In both, classes are semantically empty indices, and neither touches simulated data. We reuse CoverNet's menu as our output space and change only how each class is represented.
Semantic label embedding supplies the matching mechanism. Label-description training trains text classifiers on label definitions and gains 17–19 points over zero-shot baselines. DELE enriches labels with dictionary senses and must add cross-attention to suppress wrong senses. Both motivate us, because they show that described labels transfer across domains. DELE also constrains us. Our descriptions are generated by a deterministic program from geometry and map structure, so there is no polysemy to denoise. Phrasing sensitivity remains a risk, and we test it.
Work on the simulator gap supplies the regime and one important warning. Pre-training on Synthetic Driving Data generates 370,000 scenes with A planners on warped Argoverse maps. It reports relative gains of 8.30% minADE6 and 5.04% MR6 for DenseTNT, but it argues that supervised training on planner trajectories copies planner quirks, so it uses masked-autoencoder pretraining instead. Our proposal is supervised on synthetic decisions, so this paper directly contradicts our design, and we take it head on. Our claim is that the quirks live mostly in which exact candidate gets chosen and how often*. A relational description separates the transferable part (whether a candidate is compatible with the scene) from the non-transferable part (the simulator's choice frequencies). CoC-VLA does bridge simulation and reality through language, using a CARLA-trained teacher vision-language model whose reasoning is aligned adversarially to a nuScenes student. However, its outputs are generated text and continuous waypoints with no feasibility guarantee, and its language describes the whole scene's reasoning rather than individual candidate paths.
The closest work, Words in Motion, discretizes behavior into language-like bins ("decelerating", "turning left") and probes and steers a regression transformer's hidden states. Its "domain shift" is an artificial halving of ground-truth speed, and its bins are kinematic only. We train a classifier whose labels are the descriptions, and our descriptions are relational to map and agents, which is where we expect cross-domain invariance. LMTraj uses language for trajectories by writing coordinates as text. That costs precision and latency, 18.3 ms per trajectory against 8.2 ms. Our approach never generates text at inference: each description embedding is a table lookup plus a dot product.
Core Claim
A menu-based trajectory classifier whose classes are represented by scene-relational descriptions will lose less accuracy when moved from simulated to real traffic than the same classifier with index-based or geometry-only class representations. Most of the remaining gap can be closed by re-estimating a small per-predicate frequency correction from very little real data. The advantage should come from the relational content of the descriptions. Whether natural language is needed to encode that content, rather than a fixed attribute vector, is a separate question the experiments must answer. We expect language to matter only for predicate combinations never seen in simulation.
Research Program
The model scores candidate $k$ in scene $s$ as
$$\ell_k(s) = \frac{1}{\tau}\, g(s)^\top h\big(a_k,\, T(d_k(s))\big) + b^{\mathrm{dom}}\big(p_k(s)\big),$$
where $g(s)$ is a scene encoder over map and agent history, $a_k$ is the candidate's waypoint geometry, and $d_k(s)$ is its generated description in scene $s$. $T$ is a frozen sentence encoder, $h$ is a small fusion network, and $\tau$ is a temperature. The term $b^{\mathrm{dom}}$ is a domain-specific bias indexed by the candidate's discrete predicate signature $p_k(s)$ (for example "yield, left turn, stops at line"). We train with cross-entropy over the menu. The bias term is the only component expected to differ between simulation and reality, so with limited real data we fit it first and freeze the rest.
The description program is the main engineering artifact. For each candidate in a scene, it computes predicates against the lane graph and the other agents. The predicates include which lane the path ends in, whether it crosses a lane marking and of what type, and whether it enters a stop area or crosswalk. They also include whether it reaches a conflict point before or after each nearby agent (extrapolated at constant velocity), and its speed regime relative to the posted limit. The program renders these predicates into a sentence from templates, with several paraphrases per signature. On nuScenes we draw map semantics from the nuScenes Knowledge Graph, which encodes lane widths, marking types, stop areas and turn connectivity. That paper reports no trajectory-prediction numbers, so our pipeline doubles as a first use of its features for forecasting. On simulated scenes the same predicates come from the simulator's map. Because the predicates are defined identically in both domains, we can audit their frequencies to measure how far the simulator's driving prior is from reality.
Baselines
The naive reference points are a CoverNet-style classifier with index labels trained only on real data, and the same model trained only on synthetic data and evaluated zero-shot on real data. Their difference defines the simulator gap that every other method must shrink. The real-only model also sets the bar for whether synthetic data helps at all. CoverNet reports a 33% top-5 hit rate on nuScenes for its hybrid variant. Our reimplementation must land near that number before any comparison is trusted.
The strongest existing competitor for using synthetic trajectories is the masked-autoencoder pretraining of Pre-training on Synthetic Driving Data. It is the one method in the file shown to turn synthetic driving data into real gains, with relative improvements of 8.30% minADE6, 5.04% MR6 and 3.84% minFDE6 on Argoverse v1.1 over DenseTNT. The file gives only relative improvements, so for an absolute figure we will rely on our own reproduction. We run it on our synthetic scenes, which are regenerated with their recipe, and with our scene encoder, followed by real fine-tuning of an index-label CoverNet head. This isolates pretraining from label representation. The two are complementary, so we also report the combination.
A domain-adversarial baseline adds a gradient-reversal discriminator on $g(s)$ to the index-label classifier trained on synthetic plus unlabeled real scenes. This is the generic feature-alignment answer and the trajectory analogue of CoC-VLA's alignment. CoC-VLA itself reports ADE 1.44 and a 1.58% collision rate on nuScenes-VLM. That result is a planning setting with different inputs, so we cite it as context and do not claim to beat it.
Two ablations isolate the key ingredient. The first is a geometry-only label embedding, $h(a_k)$ with no description. It captures similarity between nearby candidates without any scene relations. If it matches our method, the descriptions add nothing beyond geometric smoothing. The second is an attribute-vector embedding that feeds the same predicates $p_k(s)$ as a binary vector instead of text. It has the same information as our method but no language. A third ablation uses kinematic-only descriptions in the style of Words in Motion bins, which tests whether relational content, not wording, does the work.
All methods share the same synthetic scenes, real training splits, scene encoder architecture, menu and training compute. Each gets the same number of hyperparameter trials, selected on a held-out real validation split. Beating the field means two things. Our method must achieve a higher top-5 hit rate and lower minADE5 on nuScenes than the best baseline at the 1%, 5% and 10% real-data budgets, with seed-level confidence intervals that do not overlap. It must also not lose at the full-data budget.
Why not just…?
Why not just regress continuous trajectories on mixed synthetic and real data? CoverNet's comparison already shows multimodal regression trailing menu classification badly on nuScenes, 13% and 10% hit rate against 33%. The synthetic-pretraining paper shows that naive supervised mixing transfers planner quirks. We include DenseTNT with masked pretraining as a reference so this claim is checked on our own splits rather than assumed.
Why not just fine-tune an index-label menu classifier on whatever real data you have? This is the most serious practical threat. With full nuScenes it may well tie our method, and we expect it to. The question is the low-data regime and the zero-shot regime, where an index classifier carries over simulated per-candidate frequencies with no structure for correcting them. The data-efficiency curves at 1–100% real data settle this. If index fine-tuning matches us at 5% real data, the proposal's practical value is small and we will say so.
Why not just use a structured attribute vector instead of language? Honestly, this may work as well on in-distribution predicate combinations, because the text is a deterministic rendering of the same predicates. Language should help only when real data contains combinations rare or absent in simulation. A frozen sentence encoder can place "yields at an unprotected left turn" near its parts, while a binary vector over unseen conjunctions has no learned weight. We test this with a held-out-composition split: we remove every synthetic scene where certain predicate combinations are the ground-truth candidate and evaluate on real scenes where they occur. If the attribute vector matches language there too, we will report the relational bottleneck as the contribution and drop the language claim.
Why not just ask a vision-language model for coordinates, or align features adversarially? LMTraj's own latency (18.3 ms per trajectory) scales poorly to dozens of agents. STSBench finds that general and driving-tuned vision-language models drop 20–30% on agent-to-agent reasoning, with DriveMM at 39.51% and OmniDrive at 29.33% accuracy. Adversarial alignment of the index model is run as a baseline. Our prediction is that it aligns scene features but leaves the simulator's per-index frequencies untouched, which the predicate-frequency audit below can show directly.
Proposed Experiments
The synthetic source regenerates the warped-map A* scenes of Pre-training on Synthetic Driving Data, which the paper reports producing at 370,000 scenes in about 20 compute hours. This gives full control over planner behavior. The real targets are Argoverse v1.1 (minADE6, minFDE6, MR6) and nuScenes (top-5 hit rate within 2 m, minADE5/10). Map predicates on nuScenes come from the knowledge-graph features. Training proceeds in four stages: synthetic only; synthetic followed by fitting the bias term $b^{\mathrm{dom}}$ on real data; synthetic plus full real fine-tuning; and real only. Real data enters at 1%, 5%, 10% and 100%, with five seeds each.
The first decisive check is the decomposition itself. On real validation data we split each error into a decision error (the chosen candidate's predicate signature differs from the ground-truth candidate's) and an execution error (same signature, different candidate). The hypothesis predicts that synthetic training with descriptions leaves mostly frequency-driven decision error. Fitting only $b^{\mathrm{dom}}$ should then remove a large share of the real-only gap with about 1% of real data, while the geometry-only and index models need far more. If fitting the bias alone recovers little, the claim that the gap is mainly a frequency gap is false.
The second check is the planner-quirk test that answers the contradicting paper. We deliberately inject a quirk into the A* planner, such as yielding with twice the real gap time, and measure whether each model reproduces it on real scenes. A description-labeled model should absorb the quirk into the yield predicate's bias, which real data then corrects. An index model should smear it across many candidates.
The third check is the held-out-composition split described above, comparing language against the attribute vector.
The fourth check covers phrasing robustness and cost. We evaluate under paraphrases of the templates never seen in training, and we report latency, since description embeddings can be cached per signature. A model whose accuracy swings across paraphrases would reproduce the prompt-sensitivity problem known from zero-shot label verbalizers. A model with negligible added latency over CoverNet would show that the language costs nothing at deployment.
References
- CoverNet: Multimodal Behavior Prediction Using Trajectory Sets
- Trajeglish: Traffic Modeling as Next-Token Prediction
- The Benefits of Label-Description Training for Zero-Shot Text Classification
- Description-Enhanced Label Embedding Contrastive Learning for Text Classification
- Pre-training on Synthetic Driving Data for Trajectory Prediction
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers
- Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction
- nuScenes Knowledge Graph - A comprehensive semantic representation of traffic scenes for trajectory prediction
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving