Vision-language models, systems that answer questions about images in text, get much worse as scenes become crowded. The usual explanation is "too many objects." This proposal tests a narrower explanation. Crowding hurts mainly because it brings separate surfaces physically close. Once surfaces are close, every step that groups things by straight-line distance starts merging them: image patches, depth-lifted point neighborhoods, and the nearest-neighbor graphs that spatial reasoners build over objects. A plate resting on another plate, a cup wedged against a box, or the two faces of a thin shelf are all near each other in space. Moving between them along the actual surfaces, however, is a long way or impossible. We propose to measure this effect directly, by holding object count fixed and only pushing objects together. We also propose a fix: when 2D image features are lifted into 3D, they may only spread along a predicted surface distance, not a straight-line one.
The opportunity comes from two literatures that have not been joined. Geometry papers have shown for decades that straight-line ("Euclidean") neighborhoods merge distinct surfaces, and they have built surface-distance ("geodesic") remedies. Those remedies only run on clean meshes of single objects or rooms. Multimodal reasoning papers have documented clutter cliffs and spatial blindness, and have answered them with more data, reinforcement learning, or chain-of-thought. None of these interventions touches how features are grouped before reasoning begins. If proximity, rather than count, drives the cliff, then the right fix sits at the perception stage. Spending more on language-side reasoning would not help much, and the evidence that extended thinking barely moves visually heavy tasks points the same way.
Core Claim
Our claim has two parts. The first is a diagnosis. Within a scene, the accuracy of spatial reasoning about objects is better predicted by the number of proximity-confusable surface pairs than by the number of objects. A proximity-confusable pair is two surface regions within a small straight-line radius whose surface-path distance is much larger.
The second part is a remedy. We lift the image to a metric point cloud. We then gate the lifting of 2D features, and the construction of the object relation graph, by a predicted intrinsic-extrinsic ratio. This removes most of that proximity-driven error, on raw single-view input with no mesh.
We borrow the intrinsic-extrinsic ratio from Meshing Point Clouds with Predicted Intrinsic-Extrinsic Ratio Guidance. For two points $p$ and $q$, it is
$$\mathrm{IER}(p,q) = \frac{d_{\mathrm{geo}}(p,q)}{\lVert p-q \rVert_2},$$
where $d_{\mathrm{geo}}$ is the shortest path along the surface. The ratio is near 1 on a continuous surface and large across a gap, a thin wall, or the boundary between two touching objects. That paper predicts the ratio with a sparse convolutional network to reject bad triangles on clean single-object scans. We instead use it as a gate on feature flow in cluttered, depth-lifted scenes. We also count scene-level high-ratio pairs as the diagnostic variable.
Research Program
The first component is a controlled clutter ladder. Existing clutter evidence confounds density with proximity. The roughly 30% chart-accuracy collapse reported by ChartMuseum and the 60-object hard split of Learning Spatial-Aware Manipulation Ordering both add objects and add contacts at the same time.
We will render synthetic indoor and tabletop scenes in a factorial design. Object count $N$ takes values from 2 to 60. A separate "compaction" factor rescales the gaps between objects from well separated to touching or wedged, holding $N$, objects, camera, and questions fixed. A third factor swaps in thin-walled or nested items such as plates, bowls, folders, and shelves. This raises proximity-confusable pairs without changing the count.
Because the renders have ground-truth meshes, we compute exact geodesics and count high-ratio pairs per scene, written $C_\tau$, using a threshold $\tau$ on the ratio. Questions follow the categories of 3DSRBench: height, location, and multi-object relations. We add support and ordering questions ("what must be removed before X?") in the style of OrderMind.
The second component is a ratio predictor that works on lifted, single-view data. We lift RGB images to metric point clouds with the depth-plus-camera-estimation pipeline of Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting. That paper itself notes edge artifacts and feature bleeding at occluding contours. On these lifted clouds we train a sparse convolutional network to predict the ratio for each nearest-neighbor edge.
Supervision comes from scenes with meshes: our renders, plus ScanNet reconstructions, where each lifted point is snapped to its nearest mesh vertex and the geodesic is computed on the mesh. This addresses the two open problems the research file attributes to VMNet and SATR: the cost of computing geodesics at inference time and the need for clean connectivity. Geodesics are needed only at training time. At test time the network predicts the ratio from local geometry in one forward pass.
The third component is the gated fusion itself. Each lifted point receives 2D features from a frozen image encoder by projection, as in PointFusion. It then refines them by message passing over its nearest neighbors. Each message is weighted by $\exp(-(\widehat{\mathrm{IER}}-1)/\sigma)$, where $\widehat{\mathrm{IER}}$ is the predicted ratio and $\sigma$ is a temperature. Neighbors across a gap or a contact boundary therefore contribute almost nothing. This is SATR's Gaussian geodesic reweighting, moved from a single "capital face" on a clean mesh to every edge of a noisy scene cloud.
Pooling these points yields object tokens. Edges between object tokens are labeled "contacting" when low-ratio bridges cross the boundary between them, and "merely near" otherwise. This replaces the pure Euclidean nearest-neighbor edges that OrderMind builds over bounding boxes. Tokens and edges are passed to a 7B multimodal model as a structured scene block, the same interface that SpatialThinker fills with model-predicted boxes. The result is a direct swap of the scene representation, with everything else held constant.
Baselines
The naive reference point is the multimodal model with image input only, no lifted geometry, and no scene block, run on the identical clutter ladder. It tells us how large the cliff is to begin with and how much of it tracks $C_\tau$ versus $N$.
The strongest reasoning-side competitor is SpatialThinker. It trains a 7B model with group-relative reinforcement learning and dense spatial rewards to write its own scene graph before answering. The file reports a "+12.1% gain" over GPT-4o on 3DSRBench but no absolute figure. We will retrain it on the same base model with the same training data and the same reinforcement learning budget. Its scene graph will then be either self-predicted or replaced by our gated block. Beating it means a statistically reliable gain on the high-compaction rungs of the ladder, with no loss on 3DSRBench as a whole.
For ordering in clutter, the strongest system is OrderMind. It reports a "95.4% success rate" on its hard 60-object split, against 71–78% for GPT-4o and Gemini-2.5 (Learning Spatial-Aware Manipulation Ordering). We will rerun it with identical upstream detections. The only change is that its Euclidean nearest-neighbor edges are swapped for our contact-labeled edges. Because 95.4% leaves little headroom on that split, the meaningful test is the high-compaction and thin-object rungs, where we expect its Euclidean graph to degrade. Beating it means a higher success rate there at matched frames per second.
For the geometry-only check, the reference is VMNet, which fuses a voxel branch with a mesh-edge branch and reports "74.6% mIoU" on ScanNet v2 against MinkowskiNet's "73.6%". We will not claim to beat VMNet on meshed input. The comparison asks how much of VMNet's mesh-based advantage our predicted-ratio gate recovers when only a lifted single-view cloud is available, with both methods trained on the same scans and given the same parameter budget.
Two ablations isolate the key ingredient. The first replaces the predicted ratio with the ground-truth ratio, which sets an upper bound. The second replaces it with a constant, which turns the gate into plain Euclidean message passing. Every comparison uses the same lifted clouds, the same frozen image encoder, and a matched hyperparameter search budget.
Why not just…?
Why not just scale the model or the pretraining data? The file's evidence is consistent: scale does not fix spatial structure. Spatial prepositions appear in "fewer than 0.2%" of LAION-2B captions (What's "up" with vision-language models?). GPT-4o scores 34.98% on ViewSpatial-Bench, where a 3B model with spatial training reaches 82.09%. We will still include the largest open model we can run on the ladder. If its compaction curve is flat, our diagnosis is wrong.
Why not just use chain-of-thought or extended thinking? ChartMuseum reports a 1.4% gain from extended thinking on visually heavy questions. That points to a failure at perception, which matches our hypothesis. We test it directly: if adding chain-of-thought closes the compaction gap while object count is held fixed, then the merging is recoverable downstream and our intervention is unnecessary.
Why not just cut lifting edges at depth discontinuities, or pool features inside segmentation masks? This is the real threat. Depth-edge cutting handles occlusion bleed cheaply. Mask-based object tokens, for example from SAM masks, prevent merging between objects that segment cleanly. We include both as baselines.
We expect the predicted ratio to win where these fail. Touching objects show no depth jump at the contact. Stacked look-alike items such as plates or folders are often merged by masks. Thin walls fall within a single mask, so the problem is within one object. The "contacting versus merely near" edge label also has no mask-based counterpart. If masks plus depth-edge cuts match our gate on the thin-object and wedged rungs, we will report that topology adds nothing beyond segmentation. The diagnosis would still stand, with segmentation as the fix.
Why not just raise voxel or point resolution? VMNet beats denser voxel networks with half the parameters, and Bilateral blue noise sampling shows that opposite faces of thin sheets interfere regardless of resolution. We will also run a resolution sweep of the Euclidean ablation to confirm that this holds on lifted clouds.
Proposed Experiments
The decisive diagnostic regresses per-question accuracy on $N$ and $C_\tau$ across the factorial ladder, for the naive model, SpatialThinker, and a frontier model. The prediction is that, at fixed $N$, accuracy falls as compaction rises. The prediction fails if the coefficient on $C_\tau$ is near zero once $N$ is controlled for.
The decisive intervention test compares the gated block against the Euclidean, mask-pooled, and depth-edge variants on the same rungs. We report spatial question accuracy, OrderMind-style picking success, and a direct bleed metric: the fraction of lifted points that inherit image features from another object's region.
For external validity, the best ladder configuration is evaluated on 3DSRBench (real and synthetic, common and uncommon viewpoints), ViewSpatial-Bench, OrderMind's Easy, Medium, and Hard splits, and CObL's TABLETOP images. TABLETOP is the one real occlusion-ordered set in the file; CObL reports 83.5% ARI there against Mask2Former's 66.3% (CObL). On TABLETOP we report support-order accuracy.
We will also report the predictor's error on the ratio itself, broken down by noise level of the lifted depth. This shows whether the gate fails gracefully when the input clouds are corrupted.
References
- Meshing Point Clouds with Predicted Intrinsic-Extrinsic Ratio Guidance
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
- Learning Spatial-Aware Manipulation Ordering
- 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
- Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting
- VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation
- SATR: Zero-Shot Semantic Segmentation of 3D Shapes
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Bilateral blue noise sampling
- CObL: Toward Zero-Shot Ordinal Layering without User Prompting