Object-Respecting Scaffolds for Dense-Scene Visual Reasoning

Vision-language models, which answer questions about images by passing visual features into a language model, break down sharply as scenes get crowded. They miscount, attach the color of one object to its neighbor, and lose track of which item is where. One surprisingly effective fix is to draw on the image. Overlaying three numbered horizontal lines and asking the model to scan band by band raises GPT-4o's counting accuracy from 12.0% to 38.8% on synthetic binding tests. The same prompt without the drawn lines does nothing. The trouble is that these lines take no account of what is in the image. They cut through objects, which produces double counts, and the method degrades beyond about 50 items (Visual Structures Help Visual Reasoning).

We propose to draw the scaffold along object boundaries instead. A segmenter first over-segments the image into fragments. A small gate then decides, for each pair of touching fragments, whether they belong to the same object, using both semantic similarity and shape cues. The resulting whole-object regions are grouped into numbered scan zones of bounded size, and the zone borders are traced only along object edges, so no object is ever cut. The model sees one full image with these borders drawn on it and scans zone by zone in a single call. The claim is that much of the remaining dense-scene failure under scaffolding comes from scaffold lines cutting objects. If that is right, removing the cuts should give gains that grow with object count, and the gains should reach attribute and relation questions, not only counting.

Core Claim

Pixel-space scaffolding helps vision-language models in clutter only to the extent that scaffold cells line up with object identity. We will test two consequences. First, accuracy gains should be predicted by the "cut rate," the fraction of ground-truth objects crossed by a scaffold line. Second, an object-respecting scaffold should keep working past the density where static lines fail, and in tightly packed scenes where there is no empty background to route a cut through.

This differs from the two closest methods in specific ways. VISER draws static lines and does no grouping. We keep its finding that the structure must be physically drawn, but we change the geometry. LVLM-Count routes cut lines through background pixels using mean-shift clustering and A* search, then crops sub-images and queries each one separately. That has three costs: latency grows with the number of crops, nothing can be compared across crop boundaries, and the method fails when objects touch, because no background path exists. Our zone borders follow merged object edges rather than empty space. We never crop, so the model keeps the global view. That view matters, since Learning to Count Anything argues that relating distant regions is what makes repetition recognizable. Keeping the whole image while imposing local structure is our practical answer to the global-versus-local tension in the file.

Research Program

The pipeline has three stages. SAM2 proposes masks that deliberately over-segment the image. Each fragment is described by DINOv2 features averaged inside its mask. This avoids re-encoding small crops, which is the resolution failure ProposalCLIP reports for CLIP on small regions.

The central difficulty is the gate. In dense scenes, two adjacent fragments with near-identical semantic features are often two instances of the same thing, such as stacked plates or neighboring emojis, rather than two halves of one object. Semantic similarity alone, the cue used by Zero-guidance Segmentation and the prototype matching in No time to train!, therefore over-merges. We combine it with geometry. For adjacent fragments $i$ and $j$, the merge probability is

$$g_{ij} = \sigma\left(w_0 + w_1 s_{ij} + w_2 b_{ij} + w_3 c_{ij}\right),$$

where the terms are as follows. $s_{ij}$ is the cosine similarity of the two fragments' mean DINOv2 features. $b_{ij}$ is the shared boundary length divided by the smaller fragment's perimeter. $c_{ij}$ is the convexity of the merged mask, measured as union area over convex-hull area. $\sigma$ is the logistic function. The intuition is that two halves of one object share a long border and form a convex union, while two touching instances share little border and form a non-convex union.

The four weights $w_0$ to $w_3$ are fit once on synthetic scenes from the binding generator, where the true object identity of every fragment is known, and then frozen. This keeps the method close to training-free while making the merge decision learned rather than a hand-set threshold, which is the limitation of ProposalCLIP's graph merging. We borrow the idea of soft gating between overlapping instances from IFAdapter, which gates overlapping instance features in a generator. Here the gate serves recognition, not generation.

Merged regions are then grouped into zones. We do this by agglomerating neighboring regions in reading order until each zone holds at most $k$ objects. The default is $k \approx 8$, chosen because FUGU shows counting falling to near zero by 16 points. Zone borders are drawn as the outer contours of the zones, which by construction follow object edges, and each zone gets a numeric label. The prompt asks the model to report per zone and then aggregate. We also include LVLM-Count's "say 0 if you see none" wording, to suppress ghost counts in empty zones.

We deliberately do not try to repair the internal hand-off from the vision encoder to the language model that FUGU diagnoses. A pixel-space intervention works with closed models and directly tests the scaffolding hypothesis. The internal fix is a separate, larger project.

Zero-guidance Segmentation constrains what the gate can do. Its part-whole failures, where a person's face and shirt stay separate, warn that the gate will sometimes leave articulated objects split. We will measure this separately as a "split rate" rather than assume it away.

Baselines

The naive reference point is the unmodified image with the task prompt. On the binding generator, VISER reports this at 12.0% counting accuracy for GPT-4o and 5.8% for Qwen2.5-VL. On FSC-147, LVLM-Count reports a GPT-4o MAE of 25.57.

The strongest scaffolding competitor is VISER itself. Its reported counting accuracy is 38.8% with GPT-4o and 40.8% with Qwen2.5-VL (Visual Structures Help Visual Reasoning). We will run it with its published three-line setup, plus a density-adapted variant whose line count is tuned on the same validation split we use to pick $k$. This gives VISER an equal tuning budget. Beating it means higher accuracy at matched model and prompt length, with the gap widening above 50 objects.

The strongest decomposition competitor is LVLM-Count. It reports GPT-4o MAE of 17.86 on FSC-147 and Qwen2-72B MAE of 24.43 on Emoji-Count (LVLM-COUNT). We will rerun it with the same VLMs. For fairness we will report total VLM calls and wall-clock latency alongside error, because our method uses one call. Beating it means equal or lower MAE overall at a fraction of the calls, and clearly lower MAE on packed-scene subsets where its A* search finds no background path. Matching it on FSC-147 while using one call would itself be a meaningful result.

A non-VLM reference is the Reference-less Class-agnostic Counter, at MAE 17.12 on FSC-147 (Learning to Count Anything). A second is simply counting our merged regions. That second baseline is the most dangerous one for counting claims.

Three ablations isolate the key ingredient. The first draws borders around every SAM2 fragment without merging. The second merges on semantic similarity only, with $w_2 = w_3 = 0$, which approximates the soft-merging criterion of No time to train!. The third uses zones with the same number of cells but random straight borders, so the number of zones is held fixed while the cut rate varies. All segmentation-based variants share the same SAM2 and DINOv2 checkpoints and the same prompt template.

Why not just…?

Why not just count the merged masks and skip the language model? For pure counting, this is a real threat. If the gate produces good instances, counting them may match or beat any VLM on FSC-147 and Emoji-Count. We include it as a baseline and will concede counting if it wins. The proposal's main claim, however, rests on questions that mask counting cannot answer: attribute, activity, and reasoning questions in VisualOverload, and the visual search, spatial relationship, and scene description tasks in the binding generator. There the scaffold has to help the VLM itself bind properties to objects.

Why not just use more or adaptively spaced straight lines, as in VISER? Straight lines must cross objects once density is high, and VISER's own analysis attributes its double counts and its decline past 50 items to exactly these crossings. The random-border ablation tests this directly. If cut rate does not predict accuracy at a matched number of zones, our explanation is wrong and adaptive straight lines would suffice.

Why not just give the model detector coordinates in text, or tell it to imagine regions? FUGU reports that injecting exact coordinates improves simple local tasks but hurts global judgments (Diagnosing Bottlenecks). VISER reports that asking the model to imagine lines yields negligible benefit. We still run a text-coordinate variant built from our merged regions, so the comparison is made with identical segmentation.

Why not just crop the zones, as LVLM-Count does? Cropping multiplies calls and removes cross-zone context. Its background-path search also fails on touching objects. The packed-scene evaluation and the latency accounting described below settle whether cropping buys anything that drawn borders do not.

Proposed Experiments

Density sweep. The first experiment regenerates binding-generator scenes at object counts from 5 to 150. On each scene we measure accuracy on all four tasks for GPT-4o and Qwen2.5-VL under every method, together with the cut rate. The decisive check is a regression of per-image accuracy gain on cut rate, controlling for object count. We predict a strong negative slope, and an accuracy gap over VISER that grows beyond 50 objects.

Counting benchmarks. The second experiment runs FSC-147 and Emoji-Count, reporting MAE, VLM calls, and latency against LVLM-Count's reported figures and our reruns. Emoji-Count is a specific test of the gate. Its visually near-identical neighboring instances are where a similarity-only merge should fail and the geometric cues should matter.

Packed scenes. The third experiment is a packed-scene probe built from Locount retail images. Ground truth comes from group boxes with instance counts. We compare per-group count error across methods. No reported VLM result on this probe was found on Lacuna, so all numbers will be our own. The prediction is that LVLM-Count degrades sharply where shelves offer no background corridor, while object-edge borders do not.

Beyond counting. The fourth experiment evaluates on VisualOverload, where the best overall accuracy is 69.5% (o3), the hardest split is 19.6%, and the best counting score is 41.7% (Gemini 2.0 Flash). We report results per category, plus logical consistency on its paired questions. We also run FUGU, where InternVL3 reaches 69.3%, to check that the scaffold does not hurt chart questions dominated by global structure.

Pass and fail criteria. The proposal succeeds if three things hold. Gains over VISER must appear on non-counting VisualOverload categories. The cut-rate relationship must hold. And the full gate must beat the similarity-only merge on Emoji-Count and the packed-scene probe. It fails if either of two things happens: the unmerged-fragment ablation matches the full method, meaning merging adds nothing, or straight borders with an equal number of zones match it, meaning object respect adds nothing.