TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route

Hongyi Luo, Qing Chen, Dihogo Gama de Matos, Hari Krishna Gadi, Yanfeng Zhang, Li Liu, Yongliang Wang, Niclas Zeller, Daniel Cremers, Lingsheng Meng2025 · DOI

Abstract: Hongyi Luo et al. (2025) introduce the TurnBack benchmark to evaluate the geospatial reasoning capabilities of Large Language Models (LLMs) through route reversal tasks. While LLMs demonstrate proficiency in landmark identification, they struggle significantly with route and survey-level knowledge. The study introduces PathBuilder, a tool to translate natural language instructions into geometric paths, revealing that even advanced models like GPT-4o and Gemini 2.5 Pro frequently fail to navigate back to a starting point. The findings suggest that current LLMs rely on superficial "word-flipping" rather than true spatial reasoning, highlighting a critical "spatial gap" in AI cognition.

The Geospatial Cognition Hierarchy

Geospatial cognition is traditionally understood through a three-tier hierarchy: Landmark knowledge (identifying specific points of interest), Route knowledge (understanding the connections and sequences between landmarks), and Survey knowledge (possessing a comprehensive "mental map" to plan novel paths).

The hierarchy of geospatial knowledge: Landmark (points), Route (sequences), and Survey (integrated maps).

While LLMs are surprisingly adept at the Landmark level due to their vast training data, they falter at the Route and Survey levels. Previous benchmarks often conflated these levels or relied on visual inputs, making it difficult to discern if a model was reasoning spatially or merely recognizing patterns. TurnBack addresses this by focusing on route reversal—a task requiring a model to take forward instructions and generate a logical return path without a global map.

Methodology: TurnBack and PathBuilder

The researchers developed a two-part framework for evaluation:

  1. TurnBack Benchmark: A set of 36,000 pedestrian routes across 12 global metropolises (e.g., Tokyo, London, São Paulo), categorized by difficulty (Easy, Medium, Hard) based on length and turn complexity.
  2. PathBuilder (PB): A novel "language-to-geometry" converter. PB acts as a bridge, translating LLM text outputs (e.g., "Turn left and walk 100m") into precise geometric coordinates using OpenStreetMap data. This allows for mathematical evaluation of the model's "mental" path.

Evaluation metrics include Return Rate (reaching the origin), Similarity Score (closeness to the optimal backtrack), and Robustness (consistency across attempts).

Experimental Results and the "Spatial Gap"

The benchmark revealed a stark performance deficit. Even on "Easy" routes (under 1km), LLMs achieved return rates between only 4.0% and 14.5%. On "Hard" routes, performance collapsed to near zero. Models frequently suffered from "disorientation," failing to identify the correct initial direction relative to the origin in 63% of cases.

Visualization of GPT-4o attempts to reverse a route, showing cascading errors and a failure to return to the starting point.

Key Insights

  • The Superficiality Trap: LLMs often resort to a "shallow symbolic flip," simply swapping "left" for "right" without reasoning about the underlying geometry. This leads to cascading errors as routes grow longer.
  • The Confidence-Accuracy Gap: Models often exhibit high token probability for directions that are geometrically impossible, indicating they are "confidently lost."
  • Representation Bottleneck: Standard tokenizers break numerical coordinates into arbitrary chunks, destroying "metric continuity" and preventing the model from understanding spatial proximity.
  • The Power of Vision: Providing a simple vector map as a visual prompt to GPT-4o boosted its return rate from 6.4% to 43.7%, suggesting that LLMs lack a stable internal spatial scaffold and rely heavily on external anchors.

Conclusion and Future Directions

True geospatial intelligence in LLMs likely requires architectural shifts beyond prompt engineering. The authors suggest three paths: developing numeric-aware tokenizers that preserve spatial relationships, implementing loss functions that penalize geometric distance rather than just string mismatches, and incorporating inductive biases (like geometric transformers) specifically suited for spatial operations like rotation and translation.

Summary updated

BibTeX
@misc{luo2025turnback,
  author = {Hongyi Luo and Qing Chen and Dihogo Gama de Matos and Hari Krishna Gadi and Yanfeng Zhang and Li Liu and Yongliang Wang and Niclas Zeller and Daniel Cremers and Lingsheng Meng},
  title = {{TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.440},
}
Figure