Ask a map assistant for "a café a short walk from the station" and the right answer depends on two things that change at different speeds. The first is the set of facts: which cafés exist, where the station entrance is, which streets are open. The second is what "a short walk" means. People stretch or shrink that phrase depending on how dense the surroundings are, so as a neighborhood is built up, the phrase covers a smaller area. Current spatial question-answering systems fix both. They query a single frozen map snapshot, and they turn vague phrases into fixed distance thresholds. Their benchmarks freeze the gold answers too.
This note proposes the first evaluation that re-scores the same fuzzy spatial questions against several dated versions of a map. It uses that evaluation to separate two kinds of decay. Fact drift is when the places change. Interpretation drift is when the meaning of the phrase changes because the surroundings changed. The note then asks where each fix should live: in the external database the model looks things up in, in a small trained module attached to a frozen language model, or in the model's own weights. Putting it in the weights carries a known cost, because fine-tuned models lose general skills.
Core Claim
Our claim is that fuzzy spatial question answering decomposes cleanly. Facts should live in a time-versioned spatial index, queried "as of" the question's date. The mapping from vague phrase to geometry should be computed from the local map context at that date, not memorized. A system built this way should track world change with zero weight updates. Two current approaches fall short of this. Large-model retrieval with fixed distance thresholds handles neither kind of drift. Fine-tuning a model on spatial data buys accuracy on one snapshot at the price of general ability.
We do not assume interpretation drift is large. Measuring its size is the first deliverable. If it turns out to be negligible, the honest conclusion is that swapping in a fresh map index is enough, and that the forgetting cost of spatial fine-tuning is never worth paying for this task.
Background and Positioning
The work on spatial language supplies the motivation. Gapp argued that spatial terms are graded rather than binary. Wu et al. model a description as a probability field over positions, built from fuzzy membership over distance and direction relative to a landmark. On Quantifying Qualitative Geospatial Data shows that these fields stretch in sparse areas and shrink in dense ones. Skoumas et al. fit mixtures of Gaussians to relations mined from 120,000 travel blogs and locate unknown places within 0.8 km in Paris. All four treat the world as static. We take their elasticity result and add time: if density changes, the field should change with it.
Retrieval for spatial questions supplies the method we start from. Spatial-RAG has a language model translate a question into spatial SQL. In doing so it estimates parameters such as "walking distance ≈ 1 km". It then scores candidates by both physical distance and text similarity, and picks among the trade-off-optimal ones with GPT-4-Turbo. MapQA builds geospatial questions from a single OpenStreetMap extract of Southern California and Illinois. GeoRAG retrieves by geographic dimension rather than plain text similarity. TP-RAG adds opening hours and visit durations for travel planning. We differ from all four in two ways: the database and the gold answers carry dates, and the distance threshold is computed from local context rather than estimated once by the model.
The temporal-decay work supplies both the evaluation template and a warning. When Benchmarks Age shows that up to 63.78% of time-sensitive BoolQ items became outdated. It also reports an Evaluation Misleading Rate above 20% for GPT-4o-mini, meaning that one in five time-sensitive questions penalized the model unfairly. Finally, it documents a "context trap": models trust stale retrieved text over correct internal knowledge. TDBench generates time-sensitive questions from temporal databases. Even there, accuracy drops 21.7% on average once the date must also be right. Both studies are limited to non-spatial facts and to one-dimensional time. We carry their drift metrics over to spatial questions, where a single change, such as a landmark closing, can silently invalidate every relation anchored to it.
The forgetting literature acts as a constraint. Model Editing Harms General Abilities shows that editing well under 1% of the weights can collapse general skills unless the updates are sparsified. Neural Knowledge Bank attaches trainable memory slots to a frozen T5. Ordinary fine-tuning drops its summarization score from 30.1 to 24.9 ROUGE-L, while the memory-slot version keeps 30.1. PIN adds a 1.2M-parameter module to a frozen vision-language model and gains object localization without hurting conversation. These results tell us that any trained spatial-interpretation component should be small and should sit outside the frozen model. PIN and the knowledge bank were also noted to have limited expressive power for relational reasoning. We test whether that limit binds for graded spatial relations.
Research Program
We first build a time-sliced version of the MapQA regions and of the TourismQA cities used by Spatial-RAG. We pair several dated OpenStreetMap snapshots of the same areas with the existing question templates. For every question we recompute the gold answer at each snapshot. Following When Benchmarks Age, we record a spatial drift score: the share of items whose answer set changes between dates. Questions anchored to fuzzy phrases ("near", "a short walk", "between") get graded gold labels instead of one hard threshold.
Those graded labels come from a modest human study of roughly two thousand items. Annotators see a dated map and rate how well a description fits a candidate place. We include a counterfactual condition that holds the target–landmark distance fixed while changing the surrounding density on the displayed map. This isolates interpretation drift from fact drift.
We then build one retrieval pipeline with three interchangeable interpreters, each turning a vague phrase plus local map statistics into a membership curve over distance and direction. The first prompts a frozen open model (Llama3.1-8B or Qwen2-7B, the backbones used by GeoRAG) with the phrase and summary density statistics. The second keeps that model frozen and trains a small head in the style of PIN or the Neural Knowledge Bank, fit to blog-derived relations as in Skoumas et al. plus the human ratings. The third applies low-rank fine-tuning to the full model on the same data. The resulting curve becomes a soft spatial filter in the SQL query. The query runs against the snapshot matching the question's date.
Baselines
The naive reference is the frozen open model answering closed-book, plus the same model with Spatial-RAG-style retrieval over the original stale snapshot. This shows how much a frozen benchmark misleads, and how often the model follows outdated retrieved records. No reported result was found on Lacuna for the spatial version of the stale-context follow rate. The closest reported figure is the over-20% misleading rate for GPT-4o-mini in When Benchmarks Age.
The strongest existing competitor is Spatial-RAG itself, with its fixed-parameter SQL generation, run over the current snapshot. It is the most dangerous baseline because it already solves fact drift by swapping the index. Spatial-RAG reports gains only relative to its own baselines: a 75.4% recall increase and a 52.6% NDCG increase on MapQA-AME, 32.0% NDCG on MapQA-ADJ, and a 19.9% Precision@1 improvement on TourismQA-NYC. No absolute scores were found on Lacuna, so we will re-run it. MapQA reports near-perfect (100%) text-to-SQL accuracy on its amenity-availability questions. This tells us those questions are not where any difference will appear, and we will report results separately by question type. For fairness, every arm uses the same backbone for SQL generation, the same snapshots, the same candidate pools and the same tuning budget. We also run one GPT-4-Turbo reference arm to anchor against the original setup.
Two ablations isolate the key ingredient. The first gives our pipeline a fixed threshold instead of a context-dependent curve. The second gives it a context-dependent curve computed from the original snapshot's density. Beating Spatial-RAG means higher time-sliced NDCG and better calibration against graded judgments on fuzzy items after drift. On rigid items we must at least match it. The fine-tuned arm must also not lose general-task performance, which we measure on a held-out suite of summarization, reasoning and tool-use tasks, scored before and after.
Why not just…?
Why not just refresh the map database and keep Spatial-RAG as is? This is the real threat. It fully handles fact drift, and if interpretation drift is small it may handle everything. The counterfactual density study and the fixed-threshold ablation settle the question directly. If graded judgments do not move with density, or if moving them changes no rankings, we will report that index refresh is sufficient and that the trained interpreter adds nothing.
Why not just tell the model the local density in the prompt and let it pick the distance? This could plausibly work. Producing one number from a density figure is far easier than the step-by-step coordinate arithmetic that Ji et al. found to degrade under chain-of-thought prompting. That is why it is a full arm of the study, not a straw man. Our expectation, not yet tested, is that it produces sensible point thresholds but poorly calibrated curves. If it matches the trained head on calibration, it wins, because it needs no training and so has no forgetting cost.
Why not just fine-tune the model on dated spatial question-answer pairs? That would place time-varying facts in the weights. Every map update would then need another update, and the editing and knowledge-bank results above show that general ability erodes with each one. The fine-tuned arm measures this cost directly. We expect it to look competitive on the snapshot it was trained on, then decay on later snapshots.
Decisive Checks
Three results decide the project. The first is the spatial drift score on time-sliced MapQA and TourismQA. If only a few percent of answers change between dates, the decay concern is overstated for spatial questions. The second is the size of the density-induced shift in human graded judgments, along with whether context-dependent curves beat fixed thresholds on post-drift fuzzy items. This is what justifies separating interpretation from facts. The third is the stale-follow rate when the model's memory, the retrieved snapshot and the true date disagree. This tells us whether the context trap reported for general facts also appears for places, and whether date-matched indexing removes it without any change to the model's weights.
References
- Basic Meanings of Spatial Relations: Computation and Evaluation in 3D Space
- On the Description-based Spatial Positioning and the Associated Uncertainty
- On Quantifying Qualitative Geospatial Data: A Probabilistic Approach
- Location Estimation Using Crowdsourced Geospatial Narratives.
- Spatial-RAG: Spatial Retrieval Augmented Generation for Real-World Spatial Reasoning Questions
- MapQA: Open-domain Geospatial Question Answering on Map Data
- GeoRAG: A Question-Answering Approach from a Geographical Perspective
- TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning
- When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation
- Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in LLMs
- Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
- Neural Knowledge Bank for Pretrained Transformers
- PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs
- Foundation models for geospatial reasoning: assessing the capabilities of large language models in understanding geometries and topological spatial relations