Separating Knowing From Pointing in Multimodal Grounding

Multimodal language models, which read images and answer in text, can often describe an object correctly and still fail to draw a box around it. Current tests cannot say why. The standard referring benchmarks, where a model must localize "the man holding the red umbrella," score only the final box. They are built from fixed public photo collections that later models have likely seen during training. A low score therefore mixes several different failures. The model may not have found the object. It may have found it but been unable to resolve the relation in the sentence. It may have resolved everything and then been unable to write the coordinates down. A high score may simply reflect memorized image-expression pairs.

We propose a regenerable grounding test that separates these causes. It crosses two things. The first is a tier of difficulty: a single isolated object, attribute disambiguation, spatial relations, and multi-hop or false-premise queries. The second is the answer format: choosing among numbered marks already drawn on the image, versus emitting a box. Because every scene is freshly composed from a known scene description, ground truth is available at every tier. We can therefore feed a model the correct answer from the tier below and measure how much of its failure is inherited from earlier mistakes. A language model writes the referring expressions from the scene description, and a symbolic checker rejects any expression that does not pick out exactly one object. The central test of the benchmark's usefulness is whether its diagnosis predicts which kind of extra training data actually fixes a given model on real photo benchmarks.

Core Claim

We hypothesize that much of the "spatial grounding gap" in current models sits at the output interface and at relational resolution, not at recognition. Stated in measurable terms, for many models the accuracy of choosing the right numbered mark will greatly exceed box accuracy even at the easiest tier. This identification-minus-localization gap will stay large at IoU 0.75, the stricter overlap threshold, even where it narrows at 0.5. We further hypothesize that part of the headline accuracy on public referring benchmarks is tied to the specific wording of their expressions.

Two quantities carry the diagnosis. For tier $k$, let $G_k = \mathrm{Sel}_k - \mathrm{Loc}_k$, where $\mathrm{Sel}_k$ is accuracy at choosing the correct mark and $\mathrm{Loc}_k$ is box accuracy at a stated IoU threshold. Let $\Delta_k = \mathrm{Acc}_k^{\text{oracle}} - \mathrm{Acc}_k^{\text{own}}$, where "oracle" means the model is given the true lower-tier answer (the correct candidate marks) and "own" means it must use its own lower-tier outputs. Under this scheme, $G_k$ measures the output-interface failure and $\Delta_k$ measures inherited error. Any failure remaining under oracle conditioning with selection answers is failure of the tier's own reasoning.

This design directly addresses the open problem that hierarchical benchmarks either assume lower tiers are solved, which is artificial, or let lower-tier errors spill into higher tiers, which confounds the diagnosis. We run both conditions on identical scenes and report the difference.

What Exists, and the Gap

Several lines of work motivate this proposal.

Two benchmarks already combine tiers with grounding, but both use fixed, manually annotated data. Point-It-Out stages embodied grounding as object localization, then affordance points, then trajectories, using about 600 samples drawn from public datasets. Dr.V sorts video hallucination into perceptive, temporal and cognitive tiers with manually annotated boxes and timestamps. We share their tiered view but add the answer-format factorization, oracle conditioning, and regeneration.

The closest four-way overlap is GeoBench. It generates uncontaminated geometry problems with a formal engine and scores four van Hiele tiers. However, its "perception" tier asks textual questions such as whether points are collinear, rather than requesting boxes. SpatialViz-Bench generates 3D spatial-visualization tasks procedurally, but in multiple-choice form. It also reports that models collapse when one task is converted to open answers, which is indirect evidence that answer format matters as much as we claim. Neither benchmark asks for coordinates.

On the grounding-method side, three existing systems bear directly on the design:

  • Rex-Thinker turns grounding into selecting among detector boxes with chain-of-thought reasoning. This is essentially the "selection" arm of our design, deployed as a system.
  • DetToolChain overlays rulers and grids on the image so the model can read coordinates off it. It helps at IoU 0.5 but not at tight boundaries.
  • Dorkenwald et al. argue that models encode position internally but lack an output channel for it.

All three are consistent with our output-interface hypothesis, but none measures it directly.

For generating expressions, we reuse two existing methods. From RIS-CQ we take the idea of writing long referring expressions from object triplets with an LLM. That work scores only overall mIoU and does not check whether the expressions are unambiguous. From INVITE we take the distort-and-verify procedure and apply it to scene descriptions to produce false-premise queries.

The dynamic-evaluation logic comes from DyVal, which regenerates text-only reasoning problems from graphs at controlled complexity. Our work carries that idea into images with continuous ground truth.

Two findings constrain the design. PhD documents "textual overrule," where models follow misleading text over visual evidence. Our false-premise tier must therefore be scored separately from grounding accuracy. Stitch and Tell reports that small models overfit to repetitive templated spatial questions, which argues against a purely template-based generator.

Research Program

Scene generator. Scenes are built by compositing segmented real-object cutouts onto real background photographs. Each scene has an explicit layer order, controlled occlusion, and controlled numbers of same-category distractors. Boxes, masks, occlusion and front/behind relations are therefore known exactly. We use real cutouts rather than CAD renders to keep photographic texture and reduce the synthetic-to-real gap.

We acknowledge that the individual cutouts may themselves appear in training data. What is new each round is the combination of objects and the referring expression.

Scene descriptions and tiers. Each scene description records objects, attributes, pairwise relations computed from geometry, and layer order. From it we generate queries at four tiers:

  • Tier 0: a single-instance category, so the model need only find and box an obvious object.
  • Tier 1: disambiguation among same-category distractors by attribute.
  • Tier 2: disambiguation by relation to another object.
  • Tier 3: two-hop relational chains, plus false-premise queries produced by distort-and-verify, where the correct response is to reject the query.

Expression generation and verification. An LLM writes expressions from the scene description. A symbolic resolver then parses each expression back into the description and discards it unless it resolves to exactly the intended target, or, for false-premise queries, to no object. A human audit on a sample estimates how often the checker passes expressions that people find ambiguous.

Evaluation protocol. Every query is posed in both the mark-selection format and the box format, under both oracle and own-output conditioning. Each evaluation round uses fresh random seeds, and a small frozen split is kept for longitudinal comparison.

A real-image test of wording memorization. Separately, on RefCOCOg and HumanRef images, the LLM paraphrases the original expressions. The same resolver-plus-human check confirms the paraphrase refers to the same object. Any score drop then reflects dependence on the original wording, with no domain shift at all. This is our cleanest contamination probe. It measures sensitivity to the exact text, which is suggestive of memorization but not proof of it.

Baselines

The naive reference point is flat end-to-end box accuracy on the public referring benchmarks, with scores on our generated scenes pooled across tiers. If our factorized measures rank models the same way as this single number and point to the same fixes, the extra machinery is unnecessary.

The strongest existing diagnostic protocol is the three-stage scoring of Point-It-Out. It is the closest tiered, coordinate-output grounding evaluation in the file. We apply its staging to our generated scenes, without our answer-format split or oracle conditioning, so the comparison uses identical images. No reported numerical scores for PIO were found on Lacuna, so it serves as a protocol baseline rather than a number to beat.

Among grounding systems, the strongest competitor is Rex-Thinker-GRPO (Rex-Thinker). It reports 86.6% average recall, 86.8% average precision, and a rejection score of 68.2 on HumanRef, against 7.1 for base Qwen2.5-VL. It is strongest here because it removes coordinate writing entirely, replacing it with detection followed by selection. It also costs 6.68s per image versus 1.13s for direct prediction.

We also evaluate DetToolChain, whose best reported figure is AP50 of 64.8 with GPT-4V on MS COCO, alongside a +24.23% gain on RefCOCO. Plain direct-regression Qwen2.5-VL is included as the model without any grounding aids.

All systems see the same scenes and queries, with the same output parsing and IoU thresholds. Prompts are tuned on a held-out generation seed with an equal number of trials per system.

The training-side baseline is Stitch and Tell, which reports +5.50% on MME-Position and +4.19% on Spatial-MM for LLaVA-v1.5-7B. It is trained with the same base model, the same number of added training samples, and the same optimization budget as our tier-targeted data.

Ablations isolate each ingredient in turn:

  • replacing LLM expressions with DyVal-style templates;
  • removing the uniqueness checker;
  • dropping oracle conditioning;
  • dropping the selection format.

Each removed ingredient should change either the diagnosis or the downstream gain. If it changes neither, that ingredient is not needed.

Beating the baselines is defined in two ways. The benchmark must produce model diagnoses that flat accuracy and PIO-style staging miss. More decisively, the diagnosis must select training data that beats both Stitch and Tell and untargeted generated data of equal size on held-out real benchmarks.

Why not just…?

Why not just break down existing static benchmarks by expression type? Per-category breakdowns of RefCOCO-style data cannot give oracle lower-tier inputs, and they cannot pose the same query in selection and box formats with guaranteed-unique referents. They also inherit the contamination risk that motivates the work. We test this directly: if a post-hoc breakdown of RefCOCOg recovers the same per-model tier ranking as our generated scenes, this alternative suffices, and we will report it.

Why not just use detect-then-select systems and declare the coordinate problem solved? This is a real threat. If Rex-Thinker-style selection closes the gap at every tier, then $G_k$ describes a problem that system design already avoids. The benchmark would then chiefly serve to show when selection is enough. Two things argue it is not always enough. First, Rex-Thinker cannot recover objects its detector misses, and our occlusion and clutter controls target exactly that case. Second, it costs roughly six times the latency. We report the system's accuracy per tier, split by whether the detector proposed the target. If it stays near oracle selection accuracy even under heavy occlusion, we will say so.

Why not just use templates instead of an LLM to write expressions? Templates are cheaper and never hallucinate. However, Stitch and Tell shows small models overfitting to templated spatial questions, and fixed wording can itself be memorized. The template ablation settles the question. If template and LLM queries yield the same model rankings and the same downstream training gains, we drop the LLM.

Why not just use a physics or CAD renderer, as SpatialViz-Bench does? Rendered scenes give perfect depth, but they widen the gap from real photos. That gap would muddy any claim that failures on generated scenes reflect real grounding ability. Compositing real cutouts keeps real textures, and the real-image paraphrase probe checks the conclusions without any domain shift.

Proposed Experiments and Decisive Checks

The first experiment evaluates at least eight open and proprietary models across all tiers, formats and conditioning modes. Metrics are box accuracy at IoU 0.5 and 0.75, selection accuracy, $G_k$, $\Delta_k$, and rejection score on false-premise queries. We also test the chain-of-thought tension recorded in the file. SpatialViz-Bench finds that step-by-step reasoning hurts open-source models, while Rex-Thinker finds it helps. We predict chain-of-thought helps at tiers 2 and 3 but does not help, or hurts, at tier 0, where the bottleneck is writing coordinates.

The second experiment is the paraphrase probe on RefCOCOg and HumanRef. A consistent drop after verified paraphrasing, larger than the drop on freshly generated scenes when their expressions are paraphrased in the same way, indicates dependence on the original wording.

The decisive check is predictive. For each of four base models, we use the diagnosis to pick which tier's generated data to fine-tune on:

  • tier-0 box data where $G_0$ dominates;
  • tier-2 relational data where oracle-conditioned tier-2 failure dominates.

We compare against two equal-size, equal-compute controls: the other tiers' data and Stitch and Tell. Evaluation is on RefCOCOg, HumanRef and MME-Position, none of which were used in generation.

The benchmark earns its place only if the diagnosis-chosen data beats both controls for most models. If untargeted data does as well, the factorization describes failures accurately but does not tell us how to fix them, and we will report that outcome.