Abstract: Hongyi Luo et al. (2025) introduce the TurnBack benchmark to evaluate the geospatial reasoning capabilities of Large Language Models (LLMs) through route reversal tasks. While LLMs demonstrate proficiency in landmark identification, they struggle significantly with route and survey-level knowledge. The study introduces PathBuilder, a tool to translate natural language instructions into geometric paths, revealing that even advanced models like GPT-4o and Gemini 2.5 Pro frequently fail to navigate back to a starting point. The findings suggest that current LLMs rely on superficial "word-flipping" rather than true spatial reasoning, highlighting a critical "spatial gap" in AI cognition.
The Geospatial Cognition Hierarchy
Geospatial cognition is traditionally understood through a three-tier hierarchy: Landmark knowledge (identifying specific points of interest), Route knowledge (understanding the connections and sequences between landmarks), and Survey knowledge (possessing a comprehensive "mental map" to plan novel paths).
While LLMs are surprisingly adept at the Landmark level due to their vast training data, they falter at the Route and Survey levels. Previous benchmarks often conflated these levels or relied on visual inputs, making it difficult to discern if a model was reasoning spatially or merely recognizing patterns. TurnBack addresses this by focusing on route reversal—a task requiring a model to take forward instructions and generate a logical return path without a global map.
Methodology: TurnBack and PathBuilder
The researchers developed a two-part framework for evaluation:
- TurnBack Benchmark: A set of 36,000 pedestrian routes across 12 global metropolises (e.g., Tokyo, London, São Paulo), categorized by difficulty (Easy, Medium, Hard) based on length and turn complexity.
- PathBuilder (PB): A novel "language-to-geometry" converter. PB acts as a bridge, translating LLM text outputs (e.g., "Turn left and walk 100m") into precise geometric coordinates using OpenStreetMap data. This allows for mathematical evaluation of the model's "mental" path.
Evaluation metrics include Return Rate (reaching the origin), Similarity Score (closeness to the optimal backtrack), and Robustness (consistency across attempts).
Experimental Results and the "Spatial Gap"
The benchmark revealed a stark performance deficit. Even on "Easy" routes (under 1km), LLMs achieved return rates between only 4.0% and 14.5%. On "Hard" routes, performance collapsed to near zero. Models frequently suffered from "disorientation," failing to identify the correct initial direction relative to the origin in 63% of cases.
Key Insights
- The Superficiality Trap: LLMs often resort to a "shallow symbolic flip," simply swapping "left" for "right" without reasoning about the underlying geometry. This leads to cascading errors as routes grow longer.
- The Confidence-Accuracy Gap: Models often exhibit high token probability for directions that are geometrically impossible, indicating they are "confidently lost."
- Representation Bottleneck: Standard tokenizers break numerical coordinates into arbitrary chunks, destroying "metric continuity" and preventing the model from understanding spatial proximity.
- The Power of Vision: Providing a simple vector map as a visual prompt to GPT-4o boosted its return rate from 6.4% to 43.7%, suggesting that LLMs lack a stable internal spatial scaffold and rely heavily on external anchors.
Conclusion and Future Directions
True geospatial intelligence in LLMs likely requires architectural shifts beyond prompt engineering. The authors suggest three paths: developing numeric-aware tokenizers that preserve spatial relationships, implementing loss functions that penalize geometric distance rather than just string mismatches, and incorporating inductive biases (like geometric transformers) specifically suited for spatial operations like rotation and translation.
Summary updated