Counterfactual-Gravity Tests for Spatial Priors in AI Models

Cognitive scientists increasingly test AI models with the same tasks they use on people, and then ask whether the model's mistakes look human. In spatial cognition this is an attractive test. People remember height more precisely than horizontal position when they walk on a floor, recall layouts best when their body is upright, and use walls to correct the drift in their sense of where they are. The trouble is that these tasks, and the human answers to them, are published. A model can reproduce the "human" error pattern by recalling the paper rather than by representing space. This is benchmark contamination: the test material, here including the expected answer, has leaked into the training data. Contamination of this kind does not inflate accuracy. It fakes a bias.

We propose a procedurally generated 3D spatial-memory benchmark whose expected error patterns come from an explicit computational observer, not from published human results. The benchmark has two kinds of trials. In canonical conditions, the observer's prediction and the published human finding agree. In counterfactual conditions, such as a viewer whose "ground" is the ceiling or a boundary that is a cliff edge instead of a wall, the two disagree. If a model tracks the published pattern where it should track the observer, it is recalling the literature. If it tracks the observer, it has something like a spatial prior. The contribution is a way to tell "has an upright, boundary-anchored spatial prior" apart from "has read about one."

Core Claim

Comparisons between human and model spatial biases are confounded by retrieval of published results. They can be made valid by generating tasks from a parametric model in which the gravity anchor and the boundary weighting are free parameters, and by scoring models in conditions where that model predicts a different pattern from the literature. Of the three input directions, geometric boundary integration supplies the observer, the postural orientation prior supplies the phenomena and the parameter under test, and contamination supplies the threat the design removes.

Background and How This Differs

The postural work gives clean, condition-specific signatures. MacNeilage et al. moved people in a motion simulator and found that heading judgments are sharper when the body is upright, even when the acceleration felt by the head is held fixed. This motivates treating "upright" as a prior and not merely an input. Hinterecker et al. rotated a virtual room and the participant's posture independently. They found that memory for a vertical 3×3 layout defaults to a body-based snapshot, so recall degrades when body and gravity verticals conflict.

Reisner et al. is the template for our main task. People memorized six objects in a 3 m virtual cube and were tested after the cube was stretched or compressed by 33%. Walkers had better vertical than horizontal precision and were fit by a Ground-Proximity model that ignores the ceiling. Joystick "flyers" showed the reverse and were fit by a Boundary-Proximity model that weights all surfaces equally. Because both the paradigm and these fits are in the literature, they are exactly what a contaminated model could parrot.

The boundary work supplies the observer we use to generate predictions. Boundary vector cells are neurons that fire when a boundary lies at a preferred distance and direction. They are modeled as a product of two Gaussian tuning curves, one over distance and one over bearing (Burgess; Barry). Place fields built from these cells stretch and shift when walls move (Foster et al.).

Two further results extend the observer to new conditions. Madl et al. treat boundaries as sensory likelihoods, so positional uncertainty grows roughly linearly with distance from the nearest boundary. Stewart et al. show that boundary cells respond to drop-offs as they do to walls, and Muessig et al. show that arena shape deforms their direction tuning, clustering it four-fold in squares but not in circles. We reuse these models as a generator of predictions. We make no claim about neurons.

The contamination work motivates the design and constrains it. Models complete classic puzzles at near-zero surprise (Yax et al.), scraped Winograd items let models skip the reasoning (Back to Square One), and creativity-test answers reflect retrieved examples (Stevenson et al.). DyVal answers contamination by generating reasoning tasks on the fly from graphs, and it found sharp drops on generated items.

We borrow DyVal's procedural generation but change what is scored. DyVal scores correctness against a ground truth. We score the shape of the errors against a model-derived prediction, because in spatial memory the bias is the object of study. No work in the file applies dynamic generation to bias-pattern evaluation. No work in the file builds counterfactual conditions in which the literature's answer and a mechanistic model's answer come apart.

Research Program

We will build an ideal observer with two named parameters. The first is a gravity-anchor vector $g$, the direction the observer treats as "down" and against which eye height serves as a ruler. The second is a boundary-weighting profile $w$ over surfaces, ranging from ground-only to all-equal. Remembered positions are posterior estimates from boundary-vector likelihoods. Precision falls with distance from weighted boundaries, following Madl et al.

From this observer we generate scenes, traversals, and deformations, and compute the predicted error anisotropy and boundary weighting for each condition. Every trial is generated with fresh object layouts, colors, sizes and phrasings, so no individual item can be memorized.

There are three task families. The first is the cube task after Reisner. Objects float in a box, and the camera either walks at a fixed eye height above a surface or flies with free pitch. Recall is tested after a 33% stretch along one axis. The second is the rotated-board task after Hinterecker. Here camera roll and scene roll are varied independently from 0° to 180°. The third is a boundary-distance task. A barrier or cliff edge is inserted, and we measure whether localization error grows with distance from it.

The counterfactual conditions change $g$ or the boundary type. A walker's anchor surface is the ceiling, with the scene rendered upside-down and all cues, including object shadows and the camera's walking bob, consistent with that. Alternatively, the only boundary is a drop-off, or gravity is stated to point sideways. In each case the observer predicts something the published human literature does not report.

Every trial exists in two formats. The rendered format shows image sequences to vision-language models. The text format describes the scene in words, the way most LLM cognitive evaluations do. Text descriptions are the most likely route to retrieval, because they can echo a paper's methods section.

Baselines

The naive reference is an isotropic-noise responder. It recalls each position with equal Gaussian error on all axes and has no boundary weighting. Any model whose error anisotropy cannot be distinguished from this responder has no measurable spatial prior, and every other comparison is moot. We fit its single noise parameter on the same held-out calibration trials used to fit every other baseline.

The second baseline is a "literature responder." It outputs the published pattern for each paradigm: walkers more precise vertically and floor-weighted, flyers the reverse, upright advantage on the board task. It applies this pattern regardless of the counterfactual manipulation. This is the precise signature of contamination. A model counts as recalling rather than representing if its errors are fit better by this responder than by the observer in counterfactual conditions.

The third baseline is the ideal observer itself in two ablated forms. One fixes $g$ to world-down and so cannot follow a ceiling anchor. The other forces $w$ to equal weights, which turns the ground prior off. These ablations isolate the key ingredient. If a model's counterfactual errors are fit equally well by the fixed-$g$ observer and the full observer, the counterfactual items are not doing their job. All observers and responders are fit by maximum likelihood with the same parameter budget of at most three free parameters, on the same trials.

The strongest existing method for contamination-resistant evaluation is DyVal-style dynamic generation, which in DyVal reports that GPT-3.5-Turbo's arithmetic accuracy drops by 23% at higher complexity, and that phi-1.5 and WizardMath-13B score nearly zero (DyVal). We implement it here as fresh-item generation of the canonical conditions only, with no counterfactuals. Our method beats it if counterfactual trials expose literature-tracking that canonical fresh items do not. Concretely, for some model, fresh canonical items show human-like anisotropy while counterfactual items show the literature responder winning over the observer. For spatial-bias benchmarks specifically, no reported result was found on Lacuna, so we report no prior number on these tasks.

Why not just…?

Why not just run a contamination detector on the paradigm descriptions? Perplexity or token-surprise tests, as in Yax et al., detect memorized text. Retrieval of a finding does not require memorized wording, though. A model can know that "walkers remember height better" from many paraphrased summaries while showing ordinary surprise on our freshly worded prompts. We will compute surprise on every text trial and report whether it predicts the contamination index. If it does predict it well, the cheaper detector suffices, and we will say so.

Why not just paraphrase or disguise the classic tasks? Surface changes keep the structure that cues the published answer: a cube, walkers versus flyers, a stretch test. DyVal's own results show that changing presentation order alone shifts performance, which suggests surface edits are a weak control. We include disguised canonical trials as an ablation. The decisive comparison is disguise versus counterfactual. Disguise alone should not separate the literature responder from the observer, because in canonical conditions they predict the same thing.

Why not just use rendered images with no text, so there is nothing to retrieve? This is the real threat. For vision-language models given only images, contamination by the literature may be negligible. In that case the interesting question is simply whether training on upright photographs yields an upright prior, analogous to MacNeilage et al.. Our design measures this directly. If the rendered and text formats give the same contamination index near zero, we conclude that contamination is a text-evaluation problem and report the image results as a clean measurement of spatial priors. Even in that case, the counterfactual-gravity trials are still needed to show that an apparent "ground prior" follows the stated anchor and is not fixed to image-down.

Proposed Experiments

The primary metric is the vertical-to-horizontal precision ratio per condition. Following Reisner et al., we also measure relative fit of ground-proximity versus boundary-proximity weighting. From these we define a contamination index: the log-likelihood advantage of the literature responder over the full observer on counterfactual trials, minus the same advantage on canonical trials. The subtraction removes any general preference a model has for one responder. Secondary metrics are the error-versus-boundary-distance slope, compared against the linear growth predicted from Madl et al., and the drop in upright advantage on the rotated-board task as roll increases.

The first decisive check is the ceiling-walker condition in text format. A model that represents the scenario should shift its weighting and precision toward the ceiling. A model that recalls Reisner et al. should stay floor-weighted.

The second check compares text and rendered formats on identical scenes. A positive contamination index in text and near zero in images would localize the problem to language-based evaluation. That pattern would show that published human-likeness claims based on text prompts need rechecking.

The third check is the drop-off condition. The observer, following Stewart et al., predicts that a cliff edge anchors position like a wall. A model that anchors only to walls reveals a learned visual association rather than a boundary representation.

We will validate the generator before testing models by running a small human sample on the rendered canonical conditions. The aim is to confirm that the observer reproduces the published human direction of effects. Without this check, disagreement with the literature responder could reflect a broken observer rather than a contaminated model.