Vision-language models, meaning language models that read images through a learned translation layer, do well when asked whether an isolated mug is fragile or a sponge is soft. They break down when the same question is asked about one item in a crowded shelf or a pile on a table. Separately, a growing body of work shows that appearance alone cannot settle many physical questions. A painted steel block looks like plastic, and a full carton looks like an empty one, so touch or sound has to break the tie. This proposal treats these two failures as a single problem, which we call physical binding. In clutter, both visual evidence and touch evidence are pooled across the whole scene, and a property belonging to one object gets reported for its neighbor.
We propose to give each object its own group of tokens before the language model sees anything. Occluded objects are filled out by a generative "amodal" completer, meaning a diffusion model (a generator that learns by gradually denoising random noise into samples) that paints in the hidden parts of each object. Every pixel carries a tag saying whether it was observed or hallucinated. Touch readings from a probe are written only into the token group of the object the probe actually contacted. The claim is testable in two sentences. First, physical-property errors in clutter are largely misbinding errors, not ignorance, so they should show up as predictions that copy a neighbor's label. Second, per-object token groups with provenance tags and contact-routed touch should flatten the accuracy-versus-object-count curve, whereas cropping, prompting tricks, or more fine-tuning data will not.
Core Claim and What Is New
The motivating evidence comes from the visual complexity literature. VLM-SubtleBench stress-tests comparisons between near-identical image pairs and finds object-existence accuracy nearly perfect at 4 objects but collapsing at 32. ChartMuseum finds that extended reasoning recovers only 1.4% for Claude-3.7-Sonnet on visually dense charts, which places the bottleneck in perception rather than logic.
On the physical side, three systems work only on single objects. Octopi-ViTaL concatenates GelSight tactile features with image features inside a language model. PhysObjects fine-tunes InstructBLIP to infer eight physical concepts from appearance. PhyGrasp bridges point clouds with physics text. None of them is tested on clutter.
The closest multi-object work takes different routes. OBEYED-VLA survives clutter by segmenting objects and converting them to depth maps, which discards the texture that carries material identity. CLAMP shows haptics can find metal in a cluttered opaque bag, but it fuses with a single late MLP. CObL decomposes tabletop scenes into occlusion-ordered, amodally completed layers but predicts no physics.
What is new here is not any single component. It is the claim, and the test of it, that three things must hold together for physical inference to survive clutter. Evidence must be bound per object. Hallucinated appearance must be kept from counting as material evidence. Touch must be routed by contact location rather than broadcast across the scene.
The provenance tag is the least obvious ingredient and deserves emphasis. A generative completer that paints the hidden half of a cup will paint a plausible material. If that painted texture is treated as observation, the model's confidence about the cup's material rises with no new information behind it. Dynamic guidance work (Mitigating Diffusion Model Hallucinations with Dynamic Guidance) already shows that generators drift toward mode-typical content, and a mode-typical texture is exactly a prior masquerading as data.
Research Program
The system has four parts.
The first is a CObL-style layer decomposer. It turns a cluttered RGB image into an ordered stack of object layers, each with a visible mask and an amodal (completed) mask. We reuse CObL's parallel-diffusion design and its synthetic pretraining recipe, whose ControlNet-depth rendering produced 2250 scenes.
The second is a per-object encoder. It runs the frozen image encoder on each completed layer and adds a learned provenance embedding to every patch token, marking the patch as observed, completed, or background.
The third is the touch router. It projects the probe's known end-effector contact point into the image and assigns tactile tokens to the object whose amodal mask contains that point. Where masks overlap at the contact, which is multi-body contact, it splits the assignment softly in proportion to layer order and mask overlap.
The fourth is a slot-structured projector. Following Deep Pre-Alignment for VLMs, a small perceiver language model sits between the encoders and the target model. DPA inserts such a model (Qwen3-0.6B) to pre-organize visual tokens and reports reduced forgetting. We constrain its attention so that tokens within one object group mix freely, while mixing across groups happens only through a small set of summary tokens. This is the projection-level analogue of the attention decoupling in AnyMS. AnyMS separates subjects during image generation using user-given boxes; we apply the separation during understanding, with inferred amodal masks.
The central diagnostic is a neighbor-capture rate. It separates misbinding from ordinary error. For a query about object $i$ with true label $y_i$ and prediction $\hat y_i \neq y_i$, let $\mathcal{N}(i)$ be the objects whose masks touch or overlap $i$. Then
$$ \mathrm{NCR} = \frac{\#\{\text{errors with } \hat y_i \in \{y_j : j \in \mathcal{N}(i)\}\}}{\#\{\text{errors}\}} \;-\; \mathrm{NCR}_{\text{chance}}, $$
where $\mathrm{NCR}_{\text{chance}}$ is the same rate computed with neighbor sets drawn at random from non-adjacent objects in the same scene. A positive, density-increasing NCR is direct evidence that clutter causes misbinding rather than general confusion.
Data
No existing benchmark pairs clutter with physical ground truth. We found no reported result on Lacuna for physical-property inference under controlled object density, so we build one in two tiers.
The synthetic tier composites objects from the PhysObjects annotations on EgoObjects, which cover 39.6K human-annotated concept labels. Scenes at densities of 1, 2, 4, 8, 16, and 32 objects mirror SubtleBench's 4-to-32 stress test, and occlusion is controlled. Per-object labels are inherited from the source annotations. Relational questions, such as which of two objects is heavier or which one contains liquid, are generated from label pairs.
The real tier arranges the 35 household objects of the Octopi-ViTaL set into tabletop piles of 2 to 16 items, since its instrument-measured hardness, elasticity, and roughness give continuous ground truth. We add deceptive pairs, meaning visually matched replicas of different material, in the spirit of PACS (PACS). A GelSight probe touches designated objects, sometimes at points where two objects are in contact.
We also evaluate layering quality on CObL's TABLETOP set. On that set CObL reports 83.5% visible-segmentation top-1 ARI, a measure of how well predicted object masks match ground truth, against 66.3% for Mask2Former. This lets us tell whether downstream failures come from decomposition.
Baselines
The naive reference point is an off-the-shelf vision-language model given the full cluttered image and a text question naming the object by description. When touch is available, Octopi-ViTaL-style concatenation of tactile tokens is appended globally to the sequence. This is what current physical-reasoning systems would do if pointed at clutter. On isolated objects this recipe reaches Spearman correlations of 0.501 for hardness, 0.530 for elasticity, and 0.643 for roughness, against a vision-only 0.307 for hardness (Octopi-ViTaL). We use the density-1 condition to check that our reimplementation reproduces numbers in this range. Then we measure the drop as density grows.
The strongest existing competitor is an object-centric crop pipeline in the style of OBEYED-VLA. It segments each object, crops it, and queries PhysObjects' fine-tuned InstructBLIP per crop, which reports 87.5% average accuracy on single-object PhysObjects test items against 66.2% for the base model (PhysObjects). For touch, the tactile reading goes with whichever crop contains the contact point. This competitor already solves binding for visible, well-separated objects, which is why it is the main threat. We run it with the same segmenter, the same fine-tuning data, and the same total tuning budget in GPU-hours as our model. We tune hyperparameters on an identical validation split with the same number of trials. Our method beats it if it achieves a smaller accuracy drop from density 1 to density 16 on per-object questions about occluded objects, higher accuracy on relational questions, and higher touch-attribution accuracy at multi-body contacts, with no loss on unoccluded single-object questions.
A second competitor is the same full-image model fine-tuned on our cluttered training scenes with matched data volume. It tests whether the gain is just exposure to clutter. Distracted Robot found that clutter fine-tuning of π0 raised overall success by 18% but reduced grasp failures by only 6%, so we expect partial but incomplete recovery.
Three ablations isolate the ingredients. The first removes the provenance embedding, so completed patches look observed. The second replaces amodal masks with visible masks, so there is no generative completion. The third replaces contact routing with global tactile tokens while keeping everything else. A fourth ablation drops the attention constraint in the perceiver, keeping DPA-style pre-alignment without slot structure. This separates the benefit of pre-alignment from the benefit of slotting.
Why not just…?
Why not just crop each object and ask a physics-tuned model about the crop? This is the real threat, and on visible, well-separated objects we expect crops to match us. Crops fail in three predictable places. An occluded object's crop contains mostly its occluder. Relational questions need both objects in one context. A probe touching two objects at once has no single crop to route to. The crop baseline is therefore run at matched budget. We report results split by occlusion fraction and question type, so that if crops match us everywhere, the proposal's slotting claim is refuted rather than obscured.
Why not just draw boxes or marks on the image and let a frontier model reason? SubtleBench reports that visual aids such as concatenation and pixel-difference overlays reduced accuracy, likely by disturbing the encoder's patch grid. ChartMuseum shows that extended thinking barely helps on dense images. Mark-based prompting is not identical to those aids, so we include a set-of-marks variant of the full-image baseline rather than assume it fails.
Why not just fine-tune on more cluttered data? Distracted Robot's 18%-versus-6% result suggests that data scaling fixes some target confusion but not the underlying failures. Our matched-data fine-tuning baseline tests this directly. If it closes the gap and its neighbor-capture rate falls to chance, the architectural claim is unnecessary.
Why not skip touch and infer physics from appearance? On deceptive replica pairs appearance is uninformative by construction. PACS reports humans rising from 90.5% to 96.3% when audio is added, and CLAMP reports 6/13 cluttered-bag retrievals with haptics against 0/13 for vision alone. The deceptive subset measures how much touch contributes once it is routed correctly.
Proposed Experiments and Decisive Checks
The first check is whether the premise holds. On the synthetic density sweep, the full-image baseline's neighbor-capture rate must be positive and rising with density. If errors in clutter are not disproportionately neighbor-copies, then binding is not the mechanism and the slot design has no reason to help. We would report that finding and stop.
The second check is the density curve itself. We plot per-object accuracy and Spearman correlation against object count for all systems. The headline result is the slope of that curve, not the value at any single density. The prediction is that the slotted model's slope is materially flatter than both the full-image and crop baselines on occluded objects.
The third check targets provenance. We evaluate on the real tier's occluded deceptive objects before touch, measuring expected calibration error (the gap between stated confidence and actual accuracy). Without provenance tags, confidence on completed regions should rise with no gain in accuracy. With tags, confidence should stay near the visible-evidence level and rise only after touch is routed in.
The fourth check is touch attribution at multi-body contacts. We measure whether the post-touch change in the model's belief lands on the contacted object rather than its neighbor. Global concatenation should split or misplace that change.
Throughout, we report decomposition quality on TABLETOP alongside the physical results. This lets us attribute any residual failure either to the layer decomposer or to the binding mechanism itself.
References
- VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
- Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference
- Physically Grounded Vision-Language Models for Robotic Manipulation
- PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Models
- Clutter-Resistant Vision-Language-Action Models through Object-Centric and Geometry Grounding
- CLAMP: Crowdsourcing a LArge-scale in-the-wild haptic dataset with an open-source device for Multimodal robot Perception
- CObL: Toward Zero-Shot Ordinal Layering without User Prompting
- Mitigating Diffusion Model Hallucinations with Dynamic Guidance
- Deep Pre-Alignment for VLMs
- AnyMS: Bottom-up Attention Decoupling for Layout-guided and Training-free Multi-subject Customization
- PACS: A Dataset for Physical Audiovisual CommonSense Reasoning
- Distracted Robot: How Visual Clutter Undermine Robotic Manipulation