Language models that answer questions about 3D rooms need to be told where things are. One recent and effective trick is to lay a coarse bird's-eye-view grid over the room, a top-down map divided into cells, and write the layout out as plain text: "cell (2,3): chair, table." A puzzling result has followed. Making the grid finer, which should give the model more precise locations, does not help and can hurt. CVP found that moving from a 6×6 to a 24×24 grid failed to improve and degraded reasoning, and MapDream found that a 384×384 navigation map costs ten times more inference than a 96×96 one for only marginal gains. Meanwhile, every task that needs to tell apart "the chair left of the desk" from an identical chair a meter away plainly needs fine spatial detail.
We propose that fine grids hurt because a uniform grid spends almost all of its text on cells the question never mentions. Resolution itself is not the problem. The fix is to let the question decide where the grid gets finer. We build a quadtree, a grid in which a cell can be split into four smaller cells. It starts from a coarse layout, and we split only the cells whose contents match the target and anchor objects named in the query. We keep the total number of text tokens fixed at roughly what the coarse grid uses. If the hypothesis is right, a token-matched, language-refined layout should beat both the coarse and the fine uniform grids. The gains should concentrate where fine detail matters: scenes with several same-category distractors, and directional relations.
Core Claim
The resolution paradox in textualized spatial layouts is a budget-allocation failure, and the budget should be allocated by cross-modal relevance. Concretely, two layouts can have the same token count, one uniform and one subdivided where query-to-object alignment is high. The second should outperform the first. A layout refined by geometric cues alone, such as point density or detector uncertainty, should recover only part of that gain.
This combines two lines of work that have not met. Adaptive subdivision methods split space on purely visual or statistical signals. PointRend refines mask points where class probability is near 0.5. Neural Sparse Voxel Fields prunes and subdivides an octree by learned density. AdaSTEM splits quadtrees by observation density. Cross-modal filtering methods do the opposite: AMLRIS masks weakly text-aligned image patches but at a fixed patch size. These methods supply our mechanism, but none uses language alignment to choose resolution.
The closest systems either serialize at a single uniform resolution, as CVP does, or compress the scene in a query-agnostic way. Video-3D LLM picks frames by greedy maximum coverage of voxel volume, so it can drop the one frame showing a small queried object. Our proposal differs from both: the spatial discretization itself is conditioned on the text.
Research Program
The base system is a 3D language model in the Video-3D LLM style. It takes RGB-D frames with 3D position encodings added to image patch tokens. We augment its prompt with a CVP-style textual layout, because CVP showed that text serialization outperforms continuous coordinate embeddings inside the language model. The layout lists, for each leaf cell, its hierarchical address, the object identifiers inside it, and one coarse height token. The height token is a cheap patch for CVP's loss of vertical information.
The refinement procedure starts from a 6×6 root grid. Each object proposal gets a relevance score $r_o = \max_{p} \cos(t_p, v_o)$. Here $t_p$ is the text embedding of a noun phrase $p$ parsed from the query, and $v_o$ is a CLIP feature of object $o$, rendered or cropped as in Multi3DRefer. A cell's score is the maximum over the objects it contains. We greedily split the highest-scoring leaf until a token budget $B$ is exhausted.
Scoring objects rather than cells matters. PointRend's known failure is that refinement cannot recover something the coarse stage missed entirely. Because we score every proposal before any splitting, a small queried object is found even if its cell is mostly empty.
We study two versions of the scorer. The first is frozen CLIP with no training. The second is a learned scorer supervised to rank cells containing the target and anchor objects highly. Sr3D's template expressions identify anchors explicitly, which provides the anchor labels.
Before building anything, a diagnostic study tests the explanation. CVP observed the resolution paradox but did not explain it. With the backbone fixed, we vary grid resolution (6, 12, and 24 cells per side) and, separately, the token budget. We do this either by deleting empty cells or by truncating to $B$. The goal is to tell three accounts apart: fine grids hurt through sheer sequence length, through the many empty cells, or through meaningless high-index cell numbers.
Baselines
The naive reference is the backbone with no textual layout at all. This is Video-3D LLM as published, which reports ScanRefer [email protected] of 58.1% and ScanQA exact match of 30.1% in Video-3D LLM. We retrain it under our schedule so that comparisons use identical data and compute.
The strongest direct competitor is the CVP-style uniform textualized grid at 6×6 and at 24×24. It embodies the current best answer to the same question. We found no reported result on Lacuna for CVP's ScanRefer, Multi3DRefer, or ScanQA numbers; the file says only that CVP outperforms Video-3D LLM. We therefore reimplement the uniform serialization inside our backbone rather than compare against published figures. Beating it means higher accuracy at equal or lower layout token count, across three seeds.
For context on non-language-model grounders, Multi3DRefer reports M3DRef-CLIP at ScanRefer [email protected] of 44.7%. DSGCR reports 70.0% on Sr3D. That is the bar for directional relations.
The key ablations hold the token budget $B$ fixed and change only how splits are chosen. One version splits randomly. One splits by point density, as in AdaSTEM. One splits by detector uncertainty, as in PointRend. One keeps a uniform 24×24 grid but deletes cells whose relevance falls below a threshold. That last version is an AMLRIS-style filter at fixed resolution. Every variant receives the same proposals, the same CLIP features, and the same hyperparameter search of four budget values. The language-guided quadtree must beat all four of these splitting rules to support the core claim.
Why not just…?
Why not just list the exact coordinates of the few relevant objects as text? This is the most serious threat. It is close in spirit to CVP's target-affinity token, which already focuses the model on query-relevant objects. If listing the top-k objects with numeric coordinates matches our method, then the hierarchy adds nothing and we will say so. We include it as a token-matched baseline. Our prediction is that it loses on relational and "furthest from" queries, where the model needs the arrangement of non-target context. Multi3DRefer reports that dropping room context in exactly this way hurts global-placement queries. We will check this on the Multi3DRefer and ScanQA subsets.
Why not just keep the coarse 6×6 grid and let the continuous 3D position encodings carry fine detail? Video-3D LLM shows these encodings matter a great deal: removing them drops Scan2Cap CIDEr from 83.76 to 31.03. Fine information may already flow through them. The no-layout and 6×6 baselines measure exactly this. If adding refined cells gives no gain on the ScanRefer "multiple" split, the encodings suffice.
Why not just refine by geometry, PointRend- or octree-style, without language? Geometric criteria refine where scenes are cluttered or ambiguous to the detector, not where the question points. The density and uncertainty ablations at matched budget test this directly. A small gap between them and our method would mean that language conditioning is not the active ingredient.
Why not just tune the backbone harder, with LoRA or LayerNorm tuning, on a fine uniform grid? Tuning LayerNorm in Attention shows that cheap adaptation can absorb much of the modality shift. To keep the comparison fair, we apply the same LayerNorm tuning to every variant. If the 24×24 uniform grid recovers under that tuning, the paradox was an adaptation problem, not an allocation problem.
Proposed Experiments
We evaluate on ScanRefer ([email protected] and [email protected], reported separately for unique and multiple splits), Multi3DRefer, Nr3D and Sr3D, ScanQA and SQA3D, and AmbiRefer3D. AmbiRefer3D tests whether refinement re-run after each clarification round improves over a static layout. Its D-LISA baseline is 23.51% on mixture ambiguity, and its proposed method reaches 41.57%.
The decisive checks are three, and each can fail. First, at matched $B$, the language-guided quadtree should beat the uniform 6×6 grid, and its gain should be concentrated on the ScanRefer multiple split and on Sr3D directional relations. A uniform gain across all splits would point to a generic prompt effect instead. Second, the frozen-CLIP scorer should beat the density, uncertainty, and random splitting rules at the same budget. Third, our method's accuracy should keep improving as $B$ grows, while the uniform grid's flattens or falls. That contrast is what would turn CVP's paradox into a measured explanation.
We will also report layout token count and inference latency for every configuration, since the whole argument concerns spending a fixed budget well.
References
- CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning
- MapDream: Task-Driven Map Learning for Vision-Language Navigation
- PointRend: Image Segmentation As Rendering
- Neural Sparse Voxel Fields
- Adaptive Spatio-Temporal Exploratory Models: Hemisphere-wide species distributions from massively crowdsourced eBird data
- AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
- Multi3DRefer: Grounding Text Description to Multiple 3D Objects
- DSGCR: Decomposed Spectral Geometry-Aware Cross-Modal Semantic Representation for 3D Visual Grounding
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning