Language models now write open-ended travel itineraries, delivery routes and field-survey plans. Other language models are increasingly asked to grade them, an approach called "LLM-as-a-judge." To grade a plan like "lunch at Café Central, then walk to the museum by 2pm," a judge must do three things. It must work out which of possibly hundreds of places in its input each name refers to. It must compute the distances and times involved. Then it must decide whether the plan holds up. Each of these is a known weak point. Frontier models get only about half of spatial-math questions right without a calculator. Long inputs quietly degrade what models can retrieve and trace. And judges justify their verdicts with rationales that hide what actually drove them.
We propose to separate the first step, deciding which real place each mention refers to, from the judging. A geospatially grounded masked language model resolves each place mention while looking only at that mention and its local map neighborhood. A masked language model here means a text encoder trained to fill in blanked-out tokens, in this case over mixed sequences of words and nearby map entities. Deterministic tools then compute distances and travel times. The LLM judge sees only this compact table of resolved evidence and never the full list of candidate places. The central empirical claim is that judge errors on long spatial contexts come mainly from grounding and retrieval, not from reasoning. If that is right, grounding first should keep agreement with human raters flat as the candidate-place context grows from 4K to 128K tokens, where flat judges fall off.
Core Claim
Our hypothesis is that an LLM judge's agreement with humans on spatial plans declines with context length, and that most of the decline comes from mis-resolved place mentions rather than bad arithmetic or bad final reasoning. Our second hypothesis is that a local-neighborhood encoder is a better tool for resolving those mentions than the judge itself, because its cost and accuracy do not depend on how long the surrounding document is.
The second hypothesis is not guaranteed, and the design tests it directly. The linking accuracy reported for existing spatial encoders is modest. GeoReasoner reports 40.1% R@1 on LGL toponym linking. It may turn out that a frontier LLM with the right prompt links places better. In that case the lasting contribution is the decomposition itself and the diagnosis of where judge errors come from, not the encoder.
Relation to Prior Work
Three lines of work each supply one piece of this proposal.
The first line supplies the grounding method. GeoLM and GeoReasoner train encoders on "spatial pseudo-sentences," in which an anchor place is followed by its nearest neighbors sorted by distance, with coordinate offsets entering through position embeddings. GeoLM doubles BERT on WikToR, reaching 32.5 versus 16.1 on the fraction of places resolved to within 20 km of the true location. Both papers evaluate only short discriminative tasks and never plug the encoder into a generation or evaluation loop. We reuse their encoders unchanged as mention resolvers inside a judge.
The second line supplies the evaluation design and also constrains it. AgentEval scores each step of an agent trace against its parent steps. It reaches a failure-detection recall of 0.89, against 0.41 for grading the final outcome only, and a Cohen's κ of 0.84 with experts. It was tested only on text and code workflows, and it assumes the judge can verify each step, an assumption that MapEval calls into question for spatial content. MapEval reports roughly 51% spatial-math accuracy without a calculator and over 85% with one. It also finds that models do worse when they must call map tools themselves than when the same information is given as text, but it never checks whether the growing tool-call trace is the cause. The Silent Judge shows that judges shift verdicts on superficial cues, by as much as +30% on ELI5, while acknowledging those cues 0% of the time. That result warns us not to trust judge rationales as evidence of what the judge checked.
The third line motivates the length axis. RULER shows that near-perfect single-needle retrieval hides failures on multi-hop tracing and aggregation, and that only about half of models claiming 32K-token contexts perform well at 32K. Resolving a place name among many same-named or nearby distractors is exactly this kind of tracing task. GRIT finds a related result for tables: flat concatenation loses information that structured integration keeps.
Research Program
We first build a length-controlled, open-ended spatial judging set seeded from MapEval-Textual. We take its place-information, routing and trip-planning scenarios and convert them to free-form requests. We collect answers from several generator models. Annotators then label each step of each answer as correct or incorrect, and each error as one of three types: wrong entity, wrong distance or time, or a constraint violation such as closed hours or an infeasible transition. This adds a spatial branch to the AgentEval failure taxonomy.
We then control context length RULER-style. The context each judge receives is padded from about 4K to 128K tokens with real OpenStreetMap place records. The padding is of three kinds: random places, near-neighbor places, and same-name places in other cities, for example a "Café Central" in both Vienna and Madrid. Placing the gold records at different depths lets us measure the drop-off for material in the middle of the context. We also build paired variants of the same plan structure set in high-coverage Western cities and in sparsely mapped cities, to measure a geographic analogue of the Silent Judge verdict-shift rate.
The proposed judge works in four stages. First, a mention extractor pulls place names out of the plan. Second, GeoReasoner or GeoLM ranks candidate entities from the context, encoding each mention together with its local pseudo-sentence rather than the whole document. Third, deterministic code computes great-circle (Haversine) distances, travel times from a routing table, and opening-hour checks. Fourth, a GPT-4o judge scores each step, AgentEval-style, using only the resolved evidence table and the step's parent steps.
We also add a check that uses the encoder as a verifier rather than a linker. We blank out each resolved place in turn and ask whether the encoder's top prediction, given the plan's neighborhood, agrees with the resolved entity. Disagreement is flagged to the judge as an uncertainty signal.
Baselines
The naive reference point is a flat judge. GPT-4o receives the full padded context and the plan, is prompted with a step-by-step rubric, and outputs a verdict for each step. It gets the same rubric, the same annotated set and the same number of prompt-tuning iterations as every other system. Every other condition has to outperform it.
The strongest existing method combines three things. The first is the flat judge given a calculator tool with Haversine and routing functions. MapEval shows this tool is what moves spatial math from about 51% to over 85%. The second is AgentEval-style step decomposition, whose reported recall is 0.89 versus 0.41 end-to-end and κ is 0.84, though on text and code rather than spatial traces. The third is lexical retrieval over the context, so that the judge sees only records whose names match mentions in the plan. We know of no reported result on Lacuna for LLM judges on open-ended geospatial itineraries, so the number to beat will be measured here. Beating it means higher step-level error recall and higher κ with human labels at 64K tokens and above, without a loss at 4K, and a slope of agreement against context length that is statistically flatter.
Two ablations isolate the key ingredient. In the first, the encoder linker is replaced by GPT-4o resolving mentions with the full context in view, with everything downstream unchanged. In the second, the linker is replaced by gold entity links, which bounds how much any linker could help. The gap between the proposed system and the gold-link oracle measures how much linking error remains. The gap between the encoder linker and the LLM linker measures whether the spatial encoder earns its place.
Why not just…?
Why not just give the judge a calculator and GIS tools? MapEval shows that tools fix arithmetic, and we include tools in both our system and the strongest baseline. A calculator cannot decide which Café Central a plan means. If the judge links the wrong entity, it computes a correct distance to the wrong place. The error-type labels let us check whether wrong-entity errors dominate at long context for tool-equipped judges. If they do not, the grounding step adds nothing, and we would report that.
Why not just let the LLM resolve places itself and then run programmatic checks? This is the real threat. The GPT-4o-linker ablation is exactly this system. RULER-style degradation predicts that full-context LLM linking gets worse as same-name distractors pile up, while the local encoder's accuracy should stay constant with context length. Its absolute accuracy may still be lower, given GeoReasoner's 40.1% R@1 on LGL. We will report where the two accuracy curves cross, if they do. If the LLM linker wins at every length, we conclude that separating grounding from judging is what matters and that the spatial encoder is not needed.
Why not just check final outcomes with unit tests? AgentEval's gap in failure-detection recall (0.41 versus 0.89) and its finding that 63% of failures propagate from upstream steps argue against outcome-only checks. We still run a programmatic outcome checker built on gold links. It tells us how much of the step-level signal a symbolic checker can recover when grounding is perfect.
Why not just use a million-token context? RULER's gap between claimed and effective context length suggests this will not be enough. The length sweep tests it directly for a long-context judge.
Proposed Experiments and Decisive Checks
The first decisive check is a plot of step-level κ with humans against context length and needle depth for each condition. The hypothesis fails if the flat judge with tools stays flat, or if its decline is not explained by wrong-entity errors in the error labels.
The second check compares the encoder linker with the LLM linker on linking accuracy and downstream κ across lengths and distractor types. It settles whether a masked-language-model spatial encoder is worth using as a judge component.
The third check measures geographic verdict shift on the paired Western and sparsely mapped plans for the flat judge and for the grounding-first judge. We also read the rationales for acknowledgment of the geographic cue, following the Silent Judge protocol. The goal is to see whether an explicit evidence table reduces geographic bias that the rationales never mention.
The fourth check measures cost as judge tokens per plan, since shrinking what the judge must read is a practical benefit of the design even if agreement ties.
Results would be reported on the new set and cross-checked against MapEval's multiple-choice accuracy, below 67% for current models with humans near 86.67%, so that judge rankings of generators can be compared with an established ground truth.
References
- GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding
- GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- RULER: What’s the Real Context Size of Your Long-Context Language Models?
- GRIT: Guided Relational Integration for Efficient Multi-Table Understanding