Do 3D Grounding Models Get the Easy Parts Right? Tier-Consistency Training for Spatial Reasoning

When a 3D language model correctly picks out "the chair between the desk and the window" in a scanned room, it is natural to assume it also knows which objects are chairs and which chair sits between those two landmarks. Nobody has checked. In text-only reasoning, ConsisEval found that models which solve a hard problem often fail an easier problem implied by it. Even GPT-4 Turbo reaches only a 92.2% consistency score, and 7B models fall to 50–70%. If the same holds for 3D grounding, then part of today's grounding accuracy comes from shortcuts, such as matching a class name when the room happens to contain one candidate, rather than from spatial reasoning. That would explain why these systems break under clutter and unusual viewpoints.

This note proposes two steps. The first is to measure this hard-to-easy inconsistency in 3D grounding cheaply. Every referring expression in ScanRefer and Nr3D can be broken into a three-tier ladder: which objects exist, how pairs of them are related, and which single object the full expression names. Because ScanNet ships with instance labels and boxes, the two lower tiers can be answered exactly from geometry, with no human or LLM annotation. The second step is to train against the inconsistency with a reward that pays the model only when its answers at the three tiers agree with one another and with the scene geometry. Contrastive alignment, meaning the step that maps 3D point features into a CLIP-style text-image embedding space, enters as the perception tier. Probing it on its own tells us whether a failure starts in the encoder or in the language model's reasoning.

Core Claim

We claim that a measurable share of correct 3D groundings are "right for the wrong reason". The model selects the correct box but answers the implied relational sub-question incorrectly. We further claim this shortcut rate is highest on exactly the queries where models are known to fail: many same-class distractors and viewpoint-dependent relations. Rewarding cross-tier agreement during reinforcement learning should lower the shortcut rate and raise accuracy on those hard slices. Simply adding the sub-questions as extra training data, or rewarding each tier separately, should not achieve the same effect, because neither ties the final answer to the reasoning that should support it. What is new is not a hierarchy of evaluation tiers. SPHERE already builds such tiers for 2D images, and shows that single-skill success fails to transfer upward to multi-skill and reasoning tasks. What is new is conditioning on success at the top tier, deriving the lower-tier labels automatically in 3D, and using agreement between tiers as the training signal.

Relation to Prior Work

Three papers motivate the gap. ScanReason separates high-level reasoning from geometric lookup through a special localization token and a "chain of grounding" through anchor objects, but it scores only final box accuracy. It therefore cannot tell whether the anchors were found correctly. Integrating Chain-of-Thought for Multimodal Alignment is the closest tiered 3D evaluation, scoring object recognition, function, and interaction separately. Its tiers, however, are judged on isolated single objects rather than multi-object scenes, and they are not paired against one another for consistency. SPHERE measures the easy-to-hard direction in 2D, while we measure the hard-to-easy direction in 3D.

Two papers supply methods we reuse. 3D-R1 trains a 3D vision-language model with group relative policy optimization, a reinforcement learning method that compares several sampled answers per prompt. Its rewards are a format reward, a 3D-overlap perception reward, and a CLIP-similarity reward. All three are flat and act on the final answer, and we add a coupling term to this recipe. StepTool showed that rewarding intermediate steps beats imitating whole trajectories in tool use, cutting tool-selection error from 12.01% to 7.86%. We carry the idea of intermediate credit into spatial grounding, where intermediate steps can be checked exactly against geometry.

Several papers constrain the design. Tool-based agents such as AgentGrounder, SPAZER, and LLM-Grounder avoid the language model's weakness at spatial arithmetic by handing geometry to deterministic tools or detectors. This shows tier-2 relations can be computed exactly, which we exploit for labels. It also means any trained model must be compared against them honestly. 3DSRBench reports 13.5% to 32.2% drops for proprietary models under uncommon viewpoints, so viewpoint-dependent relations must be treated as their own slice. On the encoder side, H2ARN argues that flat Euclidean contrastive spaces crowd part-whole hierarchies and fixes this with hyperbolic entailment cones for retrieval. SceneForge finds that Euclidean scene-text alignment degrades beyond about three objects. These results make the encoder a credible suspect for tier-1 failures in cluttered rooms, which is why we probe it separately rather than assume the fault lies in the reasoning.

Research Program

Building the tier ladder. For each ScanRefer and Nr3D utterance, a language-model parser extracts the target class, the anchor classes, and the stated relation. Each extraction is then checked against ScanNet instance labels, and parses that name absent classes are discarded.

The tiers are defined as follows. A tier-1 question asks the model to localize every instance of the target class and of each anchor class, with answers read directly from ground truth. A tier-2 question supplies the boxes of a target-candidate and anchor pair and asks whether the stated relation holds. We compute these answers geometrically for view-independent relations: near, far, above, below, larger, smaller, and between. Nr3D's view-dependence flag routes left, right, and facing relations into a separate slice, labeled only where the utterance fixes a viewpoint. Tier 3 is the original query. A random sample of 500 derived items will be audited by two annotators to estimate label noise before any result is reported.

Measuring inconsistency. The main diagnostic is the shortcut rate. It is the fraction of tier-3 successes for which the same model fails a tier-1 or tier-2 question implied by that query, and it mirrors ConsisEval's consistency score. We report it per model and per slice, sliced by number of same-class distractors, by view dependence, and by ScanReason's five reasoning types.

Locating the fault. To find where a failure originates, we run tier 1 twice. In the first run, the frozen contrastive 3D encoder retrieves instances zero-shot by class text. In the second, the full model answers. If the encoder already misses an instance, the failure belongs to alignment. If the encoder finds it and the model still fails, the failure belongs to reasoning. As a secondary arm, we swap the Euclidean encoder for an H2ARN-style hyperbolic one with scene-to-object entailment cones. This tests whether the crowding argument carries over from retrieval to cluttered-scene grounding.

Tier-consistency training. Starting from the 3D-R1 recipe on an open 3D language model, we add one term to the reward for each sampled rollout. The rollout answers the derived tier-1 and tier-2 questions along with the final query. Let $c_1, c_2, c_3 \in \{0,1\}$ mark correctness at each tier. The reward is

$$ R = R_{\text{3D-R1}} + \lambda \, c_3 \, c_1 \, c_2 - \mu \, c_3 \,(1 - c_1 c_2), $$

where $R_{\text{3D-R1}}$ is the original flat reward and $\lambda, \mu > 0$ are weights. The final term explicitly penalizes a correct box reached with wrong supporting answers. Because the tier labels come from geometry, the reward cannot be gamed by producing fluent intermediate text.

Baselines

The naive reference point is the base model trained with the unmodified 3D-R1 rewards on the same data, steps, and sampling budget. Any gain must exceed this reference. Note that the file reports 3D-R1's dense-captioning CIDEr of 91.85 but no reported result found on Lacuna for 3D-R1 grounding accuracy on ScanRefer or Nr3D, so we will measure it ourselves.

The strongest existing methods on the target benchmarks are the training-free agents. SPAZER reports 63.8% overall accuracy on Nr3D and 57.2% [email protected] on ScanRefer (SPAZER). AgentGrounder reports 52.4% on Nr3D and 41.9% [email protected] on ScanRefer at roughly 13.5 seconds per query (AgentGrounder). Among trained models, ReGround3D reports 53.1% [email protected] on ScanRefer and 30.62% on ScanReason (ScanReason).

We run released versions of each system on our tier ladder without modification, using the same Mask3D proposals where a system expects them. This yields their shortcut rates alongside their accuracies. We do not expect a single trained model to beat SPAZER's Nr3D overall figure outright. Beating it means one of two outcomes. The first is matching its accuracy on the hard and view-dependent slices with a lower shortcut rate, using a single forward pass. The second is exceeding ReGround3D and the 3D-R1 reference on ScanReason and on hard Nr3D at equal training compute.

Two ablations isolate the key ingredient, which is the coupling between tiers. The first, the sub-question data baseline, adds all tier-1 and tier-2 questions as extra supervised and reinforcement learning examples, with the same total token budget. The second, the independent-tier reward, rewards each tier's correctness separately with no coupling term. Each gets the same hyperparameter search as the main method, eight configurations apiece. The proposal wins only if coupling beats both on hard-slice tier-3 accuracy and on shortcut rate.

Why not just…?

Why not just scale the model? 3DSRBench reports that scaling gives frontier models only marginal advantages in 3D reasoning, and that their accuracy collapses under unusual viewpoints. ConsisEval shows inconsistency persists even in GPT-4 Turbo. We will still report shortcut rates across model sizes. If the rate falls toward zero with scale, our premise weakens, and the note will say so.

Why not just add the sub-questions to the training data? This is the most serious threat, because multi-task data alone might teach relational skills. It is precisely our first ablation. Our hypothesis is that the model can learn to answer sub-questions correctly in isolation while still not using those answers when grounding. Only the coupled reward targets that behaviour. If the data baseline closes the gap, the contribution reduces to the benchmark and the diagnostic, and we will report it that way.

Why not just use a tool agent that computes relations exactly? Agents such as AgentGrounder do get tier-2 right by construction. However, they inherit every detector miss and cost about 13.5 seconds per query. Our fault-location probe will show how many of their failures are unrecoverable tier-1 misses. Their shortcut rate is also informative in itself: it should be near zero on tier 2, which makes them a useful upper reference for consistency, if not for speed.

Why not just switch to a hyperbolic encoder and skip the reasoning reward? H2ARN's gains are on retrieval, raising Rsum from 220.3 to 238.5. The encoder arm tests whether these gains transfer to tier-1 grounding in clutter. If tier-1 failures turn out to dominate and the hyperbolic encoder removes them, the encoder fix matters more than the reward, and the experiments are designed to reveal which case holds.

Decisive Checks

Three results decide the question. First, the premise must hold. On hard Nr3D and ScanReason queries, the shortcut rate of the strongest trained model should be substantial, above roughly 10%, and well above its rate on easy queries. Otherwise there is nothing to train against. Second, the coupled reward must lower the shortcut rate and raise tier-3 accuracy on the hard and view-dependent slices, relative to both ablations under matched compute, across three seeds. Third, the fault-location probe should assign most residual failures to either the encoder or the reasoning stage. Encoder-dominated failures would redirect effort toward hierarchical contrastive alignment, and reasoning-dominated failures would support the reward. Either outcome is informative, and the audited, automatically derived tier ladder will be released whichever way the comparisons come out.