MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models

Overview

MapEval is a new, rigorous benchmark designed to test how well foundation models (LLMs and VLMs) handle real-world geospatial reasoning. By evaluating 30 models across textual, visual, and API-based tasks, researchers found that even the most advanced models like GPT-4o and Claude 3.5 Sonnet lag more than 20% behind human performance, particularly struggling with spatial math and multi-step trip planning.

The Challenge of Geospatial AI

While we use AI for everything from coding to creative writing, its ability to act as a "geospatial agent"—an assistant that can look at a map, use a navigation API, and plan a complex itinerary—remains surprisingly weak.

Existing benchmarks often focus on simple "where is X?" queries. However, real-world map usage is messy. It requires processing long textual descriptions of places, interacting with structured APIs to find distances, and interpreting dense visual information like road networks and map symbols. The authors address this gap by creating a comprehensive framework that forces models to reason across these different modalities (text, tools, and vision) in complex, multi-constraint scenarios.

The MapEval Framework

The researchers developed MapEval to evaluate models through three distinct lenses:

Expand

Source, p. 2

1. MapEval-Textual: Models are given long, detailed textual contexts (POI details, opening hours, distances) and must answer complex questions.
2. MapEval-API: Models act as agents using a "ReAct" framework. They must decide which map APIs to call (e.g., NearbySearch, Directions), interpret the JSON results, and iterate until they find an answer.
3. MapEval-Visual: Vision-Language Models (VLMs) are shown map snapshots and must reason about spatial relationships, such as identifying which landmark is on the "left side" of a specific route.

Crucially, the team also experimented with Tool Integration, providing models with a calculator and specific formulas (like the Haversine formula for great-circle distance) to see if offloading "spatial math" could fix common reasoning failures.

Benchmark Composition

The MapEval dataset is extensive and diverse, consisting of:

• 700 unique multiple-choice questions spanning 180 cities and 54 countries.
• Six question categories: Place Info, Nearby POIs, Routing/Navigation, Trip Planning (multi-stop), Counting (e.g., "how many hospitals?"), and Unanswerable (testing the model's ability to admit it lacks sufficient data).
• Visual Snapshots: 270 unique map images at varying zoom levels, from city-wide overviews to street-level details.
• Textual Contexts: 215 detailed descriptions generated via the Google Maps API.

Key Findings and Model Limitations

• The "Spatial Math" Wall: Models are surprisingly bad at calculating straight-line distances or determining cardinal directions (North/South/East/West) from coordinates. However, when given a calculator tool, their accuracy in these specific tasks jumped from ~51% to over 85%.
• Trip Planning is the Final Boss: This was the hardest category across the board. Planning a 3-hour museum visit followed by a 30-minute coffee break within specific opening hours proved too complex for most models to handle reliably.
• Vision vs. Reality: Models struggle with "visual clutter." As map zoom levels increase and more symbols (ATMs, bus stops, street names) appear, model performance drops significantly. They often "hallucinate" landmarks or fail to count items correctly in dense areas.
• The API Paradox: Interestingly, models often performed worse when they had to call APIs themselves (MapEval-API) compared to when the data was provided in text (MapEval-Textual). This suggests that the "agentic" overhead of choosing the right tool and parameters is still a major bottleneck.

Performance Breakdown

Expand

Source, p. 6

• Accuracy Ceiling: No model surpassed 67% accuracy on the overall benchmark.
• Top Performers: Claude 3.5 Sonnet, GPT-4o, and Gemini 1.5 Pro were the leaders. Claude 3.5 Sonnet generally held the edge in agentic (API) and textual tasks.
• The Human-AI Gap: Humans achieved ~86.67% accuracy, leaving a massive 20%+ gap that current foundation models cannot yet bridge.
• Open-Source Lag: Open-source models (like Llama-3.1 and Qwen-2.5) performed significantly worse than their proprietary counterparts, often failing to follow the complex multi-step reasoning required for trip planning.

Limitations and Future Work

The authors acknowledge several constraints in their current setup. The benchmark currently relies on five specific Google Maps APIs, whereas real-world navigation involves many more variables like traffic, weather, and public transit schedules. Additionally, the API responses are cached to ensure reproducible results, which doesn't capture the dynamic, real-time nature of actual map services. Finally, while they tested an open-ended variant, the primary benchmark is multiple-choice, which may not fully capture the nuance of a conversational navigation assistant.

Moving forward, the researchers suggest developing specialized "geospatial foundation models" and improving how VLMs handle symbolic map data (icons and legends) rather than just treating maps as standard natural images.

@inproceedings{dihan2025mapeval,
  author = {Mahir Labib Dihan and MD Tanvir Hassan and MD TANVIR PARVEZ and Md Hasebul Hasan and Md Almash Alam and Muhammad Aamir Cheema and Mohammed Eunus Ali and Md Rizwan Parvez},
  title = {{MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=hS2Ed5XYRq},
}