Relations as Dense Scans: Open-Vocabulary 3D Grounding Without an Object List

Systems that find objects in 3D scans from free-form language ("the lamp above the nightstand to the left of the bed") nearly all work the same way. They first produce a list of candidate objects, then reason over that list. When the list is missing the target or the reference object, which happens most often for rare, small, or occluded things, no amount of reasoning can recover. We propose to drop the object list from the reasoning loop. Each word-level step of a query becomes an operation on a dense "relevance field," meaning a score for every point in the scan. Category words such as "lamp" produce fields by matching each point's language-aligned feature against the text, so the vocabulary is open. Spatial words such as "above," "left of," or "farthest from" are carried out as linear-time sweeps along physically meaningful directions of the room. One such sweep is enough for any point to be influenced by any other point in the scene. Objects are extracted only once, at the very end, from the final field.

The bet is that two failures documented separately are really one problem. The first is that grounding collapses when upstream proposals miss objects. The second is that 3D language models answer spatial questions from text priors and break when the scene is rotated. Both arise because spatial relations are never computed as geometry: they are either looked up between pre-cut objects or guessed by a language model. If relations are explicit sweeps over a dense field, defined in a frame tied to gravity and the stated viewpoint, then proposal misses become recoverable, and rotation consistency follows by construction rather than having to be learned.

Core Claim

A small, typed set of relation operators can carry most of the spatial reasoning that current 3D grounding benchmarks require. The operators are directional sweeps, distance transforms, and between-ness tests. Each acts on dense open-vocabulary fields over the whole scan and costs roughly linear time in the number of occupied voxels. Composed into short programs, these operators should match object-list reasoners when proposals are good. They should clearly beat them on the cases where proposals fail, and they should be far more consistent under viewpoint rotation. The new element is not global context in a 3D backbone, and not program-based 3D reasoning. It is placing the global receptive field inside the relation operator itself, so that "global" means "along the direction the sentence names" rather than "along an arbitrary space-filling curve."

Where This Sits

Three lines of prior work motivate the proposal by exposing the gap. OpenMask3D gets open-vocabulary instance segmentation by scoring class-agnostic 3D masks one at a time against CLIP text embeddings (CLIP is a model trained to place images and their captions near each other in a shared vector space). It reaches 14.9 AP on ScanNet200 tail classes, but because each mask is scored in isolation it cannot resolve "the chair farthest from the door." Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs (APEIRIA) distills filter/relate program traces into a 3D language model and reaches 58.4% [email protected] on ScanRefer. Its authors report that it cannot recover from objects the upstream proposer misses, and that it degrades beyond three to four reasoning steps. Do 3D Large Language Models Really Understand 3D Spatial Relationships? shows that text-only models match 3D language models on standard 3D QA. On the filtered Real-3DQA set, LEO falls from 49.4% to 14.3% exact match, and top models score only 6–18% on the Viewpoint Rotation Score. We take both the proposer failure and the rotation failure as targets.

Graph-based reasoners constrain the design. ReLaGS and Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning build open-vocabulary scene graphs after the fact, using GPT-4V or fixed distance thresholds such as 0.5 m to decide relations. The latter reports that grounding lags well behind text QA. Our relations are therefore computed on the same dense substrate as the semantics, with no pairwise threshold. View-lifting reasoners such as ReasonGrounder and Reasoning3D run a 2D reasoning model on rendered views and lift its masks back to 3D, so they inherit view dependence. We keep 2D models only for producing per-point semantic features, never for relational judgments.

Two papers supply method. Mamba24/8D shows that linear recurrences (state space models, which pass a running summary along a sequence) give point clouds a scene-wide receptive field at 5.2 GB training memory and 36.3% mIoU on ScanNet200. However, it orders points along an arbitrary space-filling curve and is purely closed-vocabulary. CCNet shows that sweeps along orthogonal axes, repeated twice, let every pixel see the whole image at 11x less memory than full attention. We reuse the recurrence from the first and the axis factorization from the second. We bind each sweep direction to a spatial word, which neither paper does.

The Representation and the Operators

The substrate is a sparse voxelization of the scan, with each occupied voxel carrying a CLIP-aligned feature lifted from posed RGB frames. We lift features as OpenMask3D does with multi-view crops, and clean them with geometric distillation in the style of GeoPurify. This is where the global-local alignment gap bites. CLIP features know what a region is but bleed across boundaries, so we smooth the fields over geometric superpoints and measure the residual bleed explicitly.

A query is parsed into a short program over three operator types. filter(text) returns a field $a(x) \in [0,1]$ from the similarity between voxel feature and text embedding, followed by a calibrated sigmoid. relate(ρ, a) maps an anchor field to a new field. intersect and select combine fields and extract the final mask or box by connected components.

For a directional relation $\rho$, such as "above" or "left of," we resample the voxels into lines along the direction $d_\rho$. That direction is expressed in a frame $R$ fixed by gravity and, when the query or situation gives one, by the viewer's facing direction. We then run a learned recurrence along each line:

$$h(x) = \lambda_\rho\, h(x - \delta_\rho) + (1-\lambda_\rho)\, a(x), \qquad b_\rho(x) = \sigma\!\big(w_\rho^\top [h(x), a(x)]\big)$$

Here $\delta_\rho$ is one voxel step along $d_\rho$, $\lambda_\rho \in (0,1)$ is a learned per-relation decay that sets how far the influence reaches, $w_\rho$ is a small learned readout, and $b_\rho(x)$ is the field of points that stand in relation $\rho$ to the anchor. Lines are capped with short cross-sectional neighborhoods so that "above" does not mean "in exactly the same column." Metric relations such as "near" or "farthest from" use a soft distance transform of the anchor field, and "between" composes two transforms.

Each sweep touches every occupied voxel a constant number of times. The cost is therefore linear in occupied voxels after a sort along $d_\rho$. That cost is the price of a scene-wide receptive field that the Transformer alternative cannot pay at full resolution. Because $R$ comes from gravity and the stated viewpoint rather than from the scan's arbitrary coordinate axes, rotating the scan rotates $R$ with it, and the output field is unchanged up to discretization error. That property is exactly what the Viewpoint Rotation Score tests.

The parser that turns language into programs is trained on Sr3D-style synthetic programs, with an off-the-shelf LLM as a fallback. Parser accuracy is reported separately so that reasoning and parsing errors are not confused. Supervision is end-to-end through the soft fields: the gradient from the final grounding loss trains the relation readouts, the decays, and a light adapter on the voxel features. The CLIP features themselves stay frozen so that open-vocabulary semantics are not forgotten.

Research Program

The work has three parts. First, we build the dense executor and a stratified ScanRefer/Nr3D evaluation. In this split, each query is tagged by whether a standard proposer (Mask3D, as used by OpenMask3D) recovers the target and the anchor, and by whether the target belongs to the ScanNet200 tail. Second, we test whether learned sweeps are actually needed. The comparison is against hand-written geometric rules on the same fields: hard half-spaces for "above" and exact distance for "near." If the hand-written rules match the learned sweeps, we will report that as the finding, since it is a simpler system. Third, we study what breaks: how field quality, especially boundary bleed from CLIP features, propagates through two-, three-, and four-step programs. This tests whether soft fields avoid the step-wise collapse that APEIRIA reports for object-level chains.

Baselines

The naive reference point is direct 2D back-projection: run an open-vocabulary 2D grounder on each frame and project the result onto the point cloud. ScanRefer reports 9.04% [email protected] for its best back-projection baseline, against 22.39% for its own end-to-end model. We rerun this on our split so that the gap on proposer-miss queries is visible.

The isolated-scoring baseline is OpenMask3D with each mask matched against the full referring sentence. It isolates how much is lost by having no relational step at all.

The strongest object-level competitor is APEIRIA, which reports 58.4% [email protected] on ScanRefer, 60.5% with SegDINO3D perception, and 36.5% zero-shot accuracy on Nr3D after training on Sr3D. Alongside it we run M3DRef-CLIP, which Multi3DRefer reports at 44.7% [email protected] on ScanRefer. Fairness is enforced in three ways. All systems receive the same lifted CLIP features and the same scans. The object-level systems are given our proposer and, separately, an oracle proposer. Every learned system gets the same number of training epochs on ScanRefer/Sr3D and the same hyperparameter-search budget. Beating APEIRIA means matching its overall ScanRefer [email protected] within about two points while exceeding it by a clear margin on the proposer-miss and tail strata. On Multi3DRefer we report the zero-target and multi-target subsets, where a dense field can naturally return empty or several components. No reported Multi3DRefer number was found on Lacuna for APEIRIA, so we rerun it.

For viewpoint robustness, the reference is LEO, which scores 14.3% on Real-3DQA, and LEO with 3DR-FT, reported at 29.3%. We evaluate on the subset of Real-3DQA questions expressible in our program language and state that coverage openly. Comparisons are made on that subset by rerunning LEO there, not against the full-set numbers. Beating LEO here means a Viewpoint Rotation Score well above the reported 6–18% range at comparable accuracy.

The key ablations isolate each ingredient. Hand-written relation rules test whether learning the sweeps matters. Sweeps along a fixed space-filling curve, as in Mamba24/8D, instead of relation-bound directions test whether binding direction to language matters. The same executor restricted to fields cropped to proposer masks tests whether the dense substrate, rather than the program structure, produces the gains.

Why not just…?

Why not just run an object-level program executor on good proposals? On queries where the proposer finds both target and anchor, this is a real threat. With oracle proposals we expect APEIRIA-style executors to match or beat us, because object-level relations are cleaner than soft fields. Our claim lives on the other strata, and the stratified split settles it directly. If the proposer-miss stratum is a small fraction of ScanRefer and our gains there are small, the proposal reduces to a robustness patch, and we will say so.

Why not just attend over all objects with a Transformer? Attention over a hundred proposals is cheap, so the receptive-field argument does not apply at the object level. That is exactly why M3DRef-CLIP and APEIRIA are baselines. The issue is not cost but the object list itself: attention among proposals cannot point at an object that is not on the list. The "cropped to proposer masks" ablation shows how much of our result depends on that.

Why not give a frontier multimodal model the multi-view images? The file reports that GPT-4o scores 34.0% on VSI-Bench against 58.6% for a 3D-internalized model in Spa3R, and 22.8 against 47.0 on CA-VQA in MM-Spatial. Those are QA scores rather than grounding scores, however. We will include a GPT-4o multi-view run on a sample of our split, with answers mapped to boxes by back-projection, primarily to measure its rotation consistency.

Why not distill CLIP into a strong global backbone such as a Mamba point model and add a grounding head? This tests whether implicit global context suffices without explicit relation operators. It is the curve-ordered-sweep ablation, extended with a learned grounding head. If it matches us on relational and rotated queries, explicit operators are unnecessary. Our prediction is that it falls short on the Viewpoint Rotation Score in particular, because nothing ties its internal directions to the speaker's frame.

Decisive Checks

Three outcomes would decide the proposal. The first is [email protected] on the proposer-miss stratum of ScanRefer: the dense executor must clearly beat APEIRIA and OpenMask3D there while staying within about two points overall. The second is rotation consistency: on the expressible Real-3DQA subset and on ScanRefer scans rotated by 90°, accuracy should change by no more than discretization noise, against the 6–18% Viewpoint Rotation Score reported for current models. The third is the learned-versus-hand-written relation ablation, which determines whether the learned scans earn their place. Failure on the first check would kill the central claim. Failure on only the third would leave a simpler and still useful system.