Language models are now routinely graded by other language models, and geometry is one of the places where this is least safe. When a geometry problem includes an irrelevant measurement, a solver will often fold it into its reasoning. It may treat a stray segment as an altitude, or assume a distractor angle bisects something. It then writes a long, confident derivation that sometimes lands on the right number anyway. A model acting as a grader (an "LLM judge") that reads this derivation sees fluent, formal, correct-looking work. This proposal asks two questions. How often do judges accept such solutions? Can a judge be made to catch them by checking each geometric claim against several concrete drawings of the problem, rather than by reading the argument for plausibility?
Each ingredient has been seen separately, but not together. Reasoning models fall into a "rationalization trap" when given misleading context: they build logical bridges from false premises instead of rejecting them (Thinking in a Crowd). Geometry solvers often reach correct answers while misidentifying the underlying relationships. On Geometry3K, GPT-3.5 scored 36% on final answers but only 12% on relationship identification (Do Large Language Models Truly Understand Geometric Structures?). LLM judges reward length and formal trappings (Justice or Prejudice?). Multimodal judges that reason step by step hallucinate visual premises and use them to justify scores (MLLM-as-a-Judge). Put these together and you get a specific, measurable failure: judges certify distractor-induced, coincidentally correct geometry. That failure contaminates benchmark leaderboards and any reward signal built from judge scores.
Core Claim
We make two claims. The first is diagnostic. LLM judges grading free-form geometry solutions will accept a large share of solutions whose final answer is right but whose geometric claims are false. This false-acceptance rate will be highest when the false claim was triggered by an irrelevant given in the problem. It will also rise when the judge itself sees the distractor-laden problem rather than a clean version, even if the solution text is identical.
The second claim is constructive. A judge can be made more reliable by a procedure we call multi-realization claim checking. The judge first breaks the solution into atomic geometric claims (for example, "AD is perpendicular to BC" or "triangle ABD is isosceles"). It then writes a program that builds several numerically different figures, each satisfying only the problem's stated givens. A claim counts as valid only if it holds in every one of these figures.
The key ingredient is the plurality of figures. A coincidence, or a relationship smuggled in from a distractor, may happen to hold in one drawing. It rarely holds across randomized drawings that respect only the true constraints. This is the formal difference between "true of this picture" and "entailed by the givens."
Relation to Prior Work
Several papers motivate the gap. Thinking in a Crowd establishes the rationalization trap on science multiple-choice questions. Misleading snippets drop Qwen3-8B in thinking mode from 96.22% to 77.03%, below its 74.87% with thinking off. That work studies solvers on non-spatial questions; we study judges on spatial ones. GeomRel supplies the coincidence phenomenon, the "unrelated information" perturbation, and relationship-level ground truth, but it only evaluates solvers. PWW-Bench does use a rubric-guided Claude 3 Opus judge on diagrammatic proofs. However, it calibrates that judge on only 15 problems and never perturbs the context. Its finding that models regurgitate memorized algebra instead of reading the figure is exactly the kind of output a lenient judge would wave through.
Other papers constrain the design. MLLM-as-a-Judge finds that free-form chain-of-thought can hurt judges, and that judges often do better from text descriptions than from pixels. This argues against simply asking a judge to "reason more carefully about the figure," and for pushing verification into executed code. CALM shows that scale does not buy judging fairness and that stylistic perturbations mask subtle reasoning errors. That makes "use a bigger judge" a hypothesis to test rather than assume. MATHCHECK is the closest existing work. It crosses process judging with irrelevant disturbance on a geometry split (MATHCHECK-GEO). But it treats models as subjects answering judging questions. It does not study a judge grading open-ended solutions whose errors were caused by the distractor, and it offers no remedy.
One line of work supplies the method. Figure It Out shows that executable visual states help models generate reasoning on math benchmarks, and GeoRepEval shows that converting to a coordinate or vector form can rescue strong models. For example, Gemini 2.5 Flash on vector problems rises from 45% to 97% with convert-then-solve. We repurpose execution for verification instead of generation, and add randomization over figures, which neither paper does. Beyond Scalars separates sound from hallucinated reasoning using internal trajectories. That requires access to the model's internal states; our judge is black-box.
Research Program
Building the test set. We construct a judge-evaluation set called NoisyGeoJudge from problems with known relationship structure. The primary sources are GeomRel Basic and Advanced items. Each has a clean version and an "unrelated information" version. We add a third variant that we write ourselves: a misleading version, in which the distractor is consistent with a tempting but false relationship. We also include GeoRepEval problems in Euclidean, coordinate, and vector form, so we can test whether judges' leniency depends on representation.
Generating solutions. Solutions come from open solvers run with thinking on and off, mirroring the SciAux protocol. This yields long rationalized chains as well as short ones.
Labeling. Every solution receives two labels. The first is final-answer correctness, checked automatically. The second is geometric validity: whether every relationship the solution asserts is entailed by the givens. The validity label combines GeomRel's relationship ground truth with human annotation of the first invalid step on a stratified subset of roughly a thousand solutions. Annotators also mark whether that first invalid step uses a distractor quantity.
The resulting three-way split is valid-correct, coincidental-correct, and wrong. The coincidental-correct class is the target.
The proposed judge. The judge runs in three stages. In the first stage, an LLM rewrites the solution as a list of atomic claims, each tied to the step where it appears. In the second stage, the LLM translates only the problem's givens into a small constraint program. This program places points, enforces the stated lengths, angles, and incidences, and leaves every unconstrained degree of freedom free. A numeric solver then samples $K$ realizations by randomizing those free parameters. In the third stage, each claim is translated into a numeric predicate and evaluated in all $K$ realizations, with a tolerance. The solution's validity score is determined by its first claim that fails in any realization. That claim is reported back as a natural-language reason.
Distractor quantities enter the constraint program only as the givens they actually are. A solution that silently promotes a distractor into a structural relationship will therefore fail on the figures where that relationship is broken.
We measure formalization fidelity separately. On a held-out set, humans check whether the constraint program encodes exactly the givens. Judge errors can then be attributed either to formalization or to claim checking.
Baselines
Outcome-only grading. This is the naive reference point: compare the final answer to the reference. By construction it accepts every coincidental-correct solution. It anchors the false-acceptance scale at 100% on that class.
Rubric-guided frontier judge. This is the standard strong judge. It is a frontier model given the problem, the reference answer, and a process-judging rubric that tells it to verify every step and penalize unsupported geometric claims. It is modeled on the human-calibrated rubric judge in PWW-Bench and the process-judging task in MATHCHECK. We run it with Claude-3.5-Sonnet, which CALM found most resilient, and with GPT-4o.
Decomposition-prompted judge. This is the strongest existing method, because it attacks the same failure through prompting alone. We adapt GeoCoT's two-stage prompt to judging. The judge first enumerates the primitives and conditions in the problem, then uses reverse thinking to look for contradictions between the solution's claims and those conditions. As a solver aid, GeoCoT improves GeomRel accuracy by 9.15 points on Basic and 14.79 on Advanced. We found no reported result on Lacuna for any judge's false-acceptance rate on coincidentally correct geometry solutions. We found no such result for MATHCHECK-GEO process-judging accuracy either. The baseline numbers must therefore be measured here.
Fairness controls. Every judge uses the same underlying model, sees the same inputs, and gets the same total token budget, with our method's code-generation calls counted against that budget. Each judge also receives the same prompt-tuning effort: a fixed number of prompt revisions on a shared development split.
Ablations. Two ablations isolate the key ingredient. In the first, we set $K=1$, checking claims on a single realization. This tests whether randomization rather than mere execution does the work. In the second, the same LLM extracts the claims but judges each one by itself without running code. This tests whether execution matters at all.
What counts as winning. Beating the strongest baseline means two things. Our judge must cut false acceptance on coincidental-correct solutions by a margin outside bootstrap confidence intervals. It must also keep acceptance of valid-correct solutions within a few points of the rubric judge.
Why not just…?
Why not just strip the distractors before judging? Context filtering works in retrieval settings. Distraction-aware answer selection raises exact match by 64% on average (Improving Zero-shot Reader), and CompAct compresses context 37–51x. This is a real threat for the judge-side half of our claim, so we test it directly. One condition gives the judge an LLM-filtered problem statement; an oracle condition gives it the clean GeomRel version.
Filtering cannot repair the solution side, though. The flawed claim is already in the student's text, and a judge that sees only clean givens must still notice that "BD is an altitude" was never stated. If oracle-clean judging closes most of the gap, we will report that the problem is judge-side distraction rather than judge-side spatial checking, and narrow the contribution accordingly.
Why not just use a stronger judge with careful chain-of-thought? MLLM-as-a-Judge finds that judges reasoning step by step hallucinate premises and score worse. Thinking in a Crowd finds that thinking mode amplifies misleading context. CALM finds that capability does not predict judging fairness. Each of these, however, was observed in a different setting. We therefore run the rubric and decomposition judges with thinking on and off, across model sizes, and report whether the gap closes with scale.
Why not just formalize the solution and hand it to a symbolic geometry prover? This would be strictly better if free-form solutions could be formalized reliably. They usually cannot: they mix prose, algebra, and implicit figure assumptions. A full proof-checker also rejects valid but informal solutions. Our method formalizes only the givens, which are short, and checks claims numerically. This trades soundness for coverage. The formalization-fidelity measurement shows how much that trade costs. Where a prover-ready formalization of the solution exists, we will compare against it on that subset.
Why not just compare final answers across several problem variants? Point relabeling and representation swaps catch solvers whose answers are unstable, as in GeoRepEval. Coincidental solutions, however, can be stably wrong in their reasoning while stably right in their answer, so answer consistency cannot flag them. We report this baseline on the GeoRepEval portion as a cheap competitor.
Proposed Experiments
The first decisive check is the diagnostic. We measure each baseline judge's false-acceptance rate on coincidental-correct solutions, stratified in three ways: whether the first invalid step used a distractor, whether the solver ran with thinking on or off, and how long the solution is. The hypothesis predicts higher acceptance for distractor-triggered and longer chains. If acceptance is flat across these strata, the rationalization story does not transfer to judging.
The second check is causal on the judge side. We hold each solution fixed and swap only the problem statement the judge sees: clean, unrelated-information, or misleading. Any change in verdict is attributable to the judge's own sensitivity to noise.
The third check is the method comparison. Agreement with the human validity labels is measured as balanced accuracy over valid versus invalid. We pair this with the false-acceptance and false-rejection rates, and with the two ablations. The predicted signature is that $K=1$ recovers only part of the gain, and that the no-execution ablation recovers little. If $K=1$ matches the full method, the "many figures" idea is unnecessary and the contribution reduces to executed claim checking.
The fourth check is about representation. On GeoRepEval, we ask whether judge leniency differs across Euclidean, coordinate, and vector forms of the same solution. The proposed judge should be largely invariant, because it always verifies in coordinates.
The last check is external. We run the judge on the geometry portion of the MLLM-as-a-Judge benchmark and on MATHCHECK-GEO process-judging items. The judge receives text descriptions of the figures, the input format MLLM-as-a-Judge found most reliable. We compare against the reported GPT-4V reference points from that benchmark (Pearson 0.490 for scoring, about 79% human agreement for pair comparison), while noting that those figures cover all domains, not geometry alone.
References
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Do Large Language Models Truly Understand Geometric Structures?
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- PWW-Bench: Probing Visual Mathematical Reasoning with Proofs Without Words
- Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
- Figure It Out: Improve the Frontier of Reasoning with Executable Visual States
- Measuring Representation Robustness in Large Language Models for Geometry
- Beyond Scalars: Evaluating and Understanding LLM Reasoning via Geometric Progress and Stability
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- CompAct: Compressing Retrieved Documents Actively for Question Answering