TrajPrism: A Multi-Task Benchmark for Language-Grounded Urban Trajectory Understanding

Lihuan Li, Wilson Wongso, Baiyu Chen, Hao Xue, Ruiyi Yang, Yifan Duan, Xiachong Lin, Yang Song, Flora SalimarXiv 2026 · arXiv

TL;DR

TrajPrism is a new large-scale, multi-task benchmark designed to bridge the gap between spatial urban mobility and natural language understanding. By pairing 300K real-world GPS trajectories across three major cities with 2.1 million language instances, it challenges models to perform bidirectional tasks: generating trajectories from instructions, retrieving trajectories via semantic queries, and captioning routes. The paper reveals that traditional, geometry-only mobility models fail significantly when language is introduced into the interface, establishing language-trajectory alignment as an open challenge.

What Problem Does This Paper Address?

Urban mobility analysis has traditionally been split into two isolated domains. On one side, trajectory modeling is highly geometry-centric, focusing on coordinates, distances, and shapes while ignoring human intent and context. On the other side, recent language-centric mobility benchmarks evaluate large language models (LLMs) on high-level route planning or tool use, but they lack fine-grained, verifiable alignment with actual physical routes.

What is missing is a unified framework that evaluates both modalities together on real-world data. To address this, the authors introduce TrajPrism, a benchmark that grounds unconstrained natural language directly into real urban trajectories, forcing models to understand travel intents, constraints, and preferences.

Figure 1: TrajPrism refracts real urban trajectories through a four-dimensional intent taxonomy into diverse, grounded language annotations. The benchmark supports three complementary tasks: navi- gation instruction following, trajectory retrieval, and trajectory captioning, across 300K trajectories and three cities.

Methods

The authors developed a reproducible data generation pipeline called Reverse Intent Reconstruction (RIR) to synthesize high-quality language annotations from raw GPS traces:

  1. Phase-based Semantic Route Abstraction: Raw trajectories are compressed into semantic "phases" using Uber’s H3 hexagonal hierarchical spatial index. This run-length encoding aggregates headings, durations, street names, and nearby Points of Interest (POIs).
  2. Intent Sampling: Travel intent is sampled from a four-dimensional taxonomy: Destination, Waypoint, Route Preference, and Temporal/Pace.
  3. Multi-task Generation: Using a mix of Gemini 3.1 Flash, Claude 4.6 Sonnet, and GPT-OSS 120B, the pipeline generates three instruction variants (Literal, Concise, Chatty), three retrieval queries, and one factual caption per trajectory.
  4. Quality Control: A five-stage deterministic filter is paired with a cascaded LLM-and-human judging process to eliminate hallucinations and ensure spatial accuracy.
Figure 2: Dataset generation pipeline. Steps 1–4 constitute the Reverse Intent Reconstruction (RIR) framework: map-matched trajectories are compressed into grounded semantic phases, travel intent is reconstructed via structured sampling over a four-dimensional taxonomy, and multi-task language an- notations are synthesized across three tasks. The generated data then undergoes five-stage quality con- trol (Step 5) and cascaded LLM and human judging (Step 6), yielding the final TrajPrism benchmark.

To calibrate the benchmark, the authors also introduced three baseline proof-of-concept models:

  • TrajAnchor (Generation): A RAG-based pipeline that retrieves a similar trajectory as a spatial anchor, extracts constraints using an LLM, and routes the path via a soft-weighted chain Dijkstra search.
  • TrajFuse (Retrieval): A dual-encoder contrastive framework that fuses geometric trajectory embeddings with H3 semantic text descriptions, aligning them with text queries using an InfoNCE loss.
  • TrajRap (Captioning): A retrieval-augmented few-shot prompting pipeline that generates factual captions from structural features and semantic context.
Figure 3: TrajAnchor pipeline (Task 1). Step 1 retrieves similar training trajectories; Step 2 extracts and grounds spatial constraints via LLM; Step 3 generates the route via chain Dijkstra. Bottom: Porto example with ground truth on the left.

Datasets

The newly introduced TrajPrism benchmark consists of:

  • 300K real-world trajectories sampled across three distinct cities: Porto, San Francisco, and Beijing.
  • 2.1M total task instances (7 annotations per trajectory).

Unlike existing benchmarks like Semantic Routing (which uses synthetic queries) or MobilityBench (which focuses on API tool use for agent planning), TrajPrism is the only dataset built from real GPS trajectories that supports bidirectional language-to-trajectory and trajectory-to-language tasks.

Key Insights and Ideas

  • Geometry-Only Models Fall Short: Purely geometric trajectory encoders (like UniTraj or BLUE) completely fail at language-grounded tasks because they cannot capture abstract human preferences or temporal constraints.
  • Task-Specific Bottlenecks:
  • Generation is primarily bottlenecked by destination selection. When an oracle provides the correct destination, trajectory fidelity (Jaccard similarity) nearly doubles.
  • Retrieval struggles heavily with implicit temporal or preference-based queries (e.g., "avoid high traffic"), whereas explicit spatial queries are handled much better.
  • Captioning models are highly fluent but suffer from poor spatial grounding; they struggle to correctly recall specific POIs without direct access to map-level semantic context.
  • Map-Level Semantics as an Enabler: Integrating structured spatial knowledge (like H3 cell area descriptions) consistently boosts performance across all three tasks, serving as the necessary bridge between raw coordinates and natural language.

Key Results

  • Task 1 (Trajectory Generation): Even the best-performing TrajAnchor variant achieved only a 17–20% Destination Hit Rate and a low Jaccard similarity (~0.25–0.30) across cities, underscoring the difficulty of the task.
  • Task 2 (Semantic Retrieval): TrajFuse outperformed all text-only and geometry-only baselines. However, its Hard Recall@1 remained below 0.10 in Porto and below 0.23 in other cities, leaving massive headroom for improvement.
  • Task 3 (Trajectory Captioning): While language fluency scores were high (BERTScore F1 > 0.83), structural-only baselines achieved a dismal POI Recall of just 0.015 in San Francisco. Adding semantic context (TrajRap) dramatically improved POI Recall to 0.341 in Porto.

Limitations and Future Work

The authors note several limitations that pave the way for future research:

  • Language and City Diversity: The benchmark currently features English-only annotations and covers only three cities.
  • Synthesis Bias: The travel intents are LLM-synthesized, which might introduce distributional biases compared to organic, human-generated user queries.
  • Scope Constraints: TrajPrism is restricted to map-matched vehicle GPS traces. It does not yet address indoor navigation, pedestrian routing, or multimodal transportation networks.

Summary updated

Abstract

Urban mobility is naturally expressed both as trajectories in space and as natural-language descriptions of travel intent, constraints, and preferences. However, prior work rarely evaluates these two modalities together on the same real-world trajectories: trajectory modeling often stays geometry-centric, while language-centric mobility benchmarks frequently target route planning and tool use rather than fine-grained, verifiable alignment between text and the underlying route. We introduce TrajPrism, a multi-task benchmark for language-trajectory alignment that unifies (i) instruction-conditioned trajectory generation, (ii) language-driven semantic trajectory retrieval, and (iii) trajectory captioning, together with an evaluation protocol that measures trajectory fidelity, retrieval quality, and language groundedness. We construct TrajPrism by pairing real urban trajectories with judge-filtered language annotations generated under a four-dimensional travel-intent taxonomy. The benchmark contains 300K selected trajectories across Porto, San Francisco, and Beijing, yielding 2.1M task instances from three instruction variants, three retrieval queries, and one caption per trajectory. We further develop proof-of-concept models for each task: TrajAnchor for instruction-conditioned trajectory generation, TrajFuse for semantic trajectory retrieval, and TrajRap for trajectory captioning. These models instantiate the proposed tasks and show that geometry-only trajectory baselines leave a large gap on our protocol, especially where language is part of the input-output interface. We release TrajPrism with code and a reproducible annotation pipeline that is designed to be portable across cities, given compatible trajectory inputs and map resources.
BibTeX
@misc{li2026trajprism,
  author = {Lihuan Li and Wilson Wongso and Baiyu Chen and Hao Xue and Ruiyi Yang and Yifan Duan and Xiachong Lin and Yang Song and Flora Salim},
  title = {{TrajPrism: A Multi-Task Benchmark for Language-Grounded Urban Trajectory Understanding}},
  year = {2026},
  eprint = {2605.10782},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.10782},
}
Figure