Seams Through the Gaps: Diffusion-Guided Visual Scaffolds for Dense-Scene Reasoning

Vision-language models, the systems that answer questions about images in text, break down sharply once a scene holds dozens of objects. A surprisingly effective fix is to draw a few labeled horizontal lines on the image and tell the model to scan it strip by strip. The fix has a flaw: straight lines cut through objects. A person sliced in half gets counted twice, or has their attributes split across two strips. We propose drawing the lines so they bend around objects, following the gaps between them. To find those gaps we use a classical, training-free tool. The image is broken into small regions, the regions are connected into a graph, and relevance is diffused across that graph. Pairs of regions that diffusion ties together strongly are treated as parts of the same object, and the lines are forbidden from separating them.

The point matters because the obvious remedies have been tested and fail. Higher resolution, larger models and step-by-step text reasoning all leave the dense-scene collapse largely intact. Avoiding cut objects currently requires chaining a detector and a segmenter, a pipeline that is slow and needs a named object category. A seam computed from graph diffusion in a fraction of a second would make the scaffold idea content-aware without these costs. Because the scaffold is just lines drawn on the image, it also applies to open-ended questions, not only counting.

Core Claim

The gains that scaffolds give in dense scenes are limited mainly by how many objects the scaffold fragments. Fragmentation can be minimized without any detector by routing each scaffold line along a minimum-cost path whose cost comes from diffusion affinity on a region graph. We further claim that the standard way graph saliency methods choose their starting points, treating the image border as background, is the wrong signal for dense scenes, because there the "foreground" fills the frame. A pairwise diffusion affinity that asks "do these two regions belong together?" should beat a foreground/background saliency map that asks "is this region salient?"

Background and Positioning

Our method comes from the graph saliency literature. Saliency Detection via Graph-Based Manifold Ranking segments the image into superpixels (small regions of similar color) and links them in a graph whose borders are joined in a loop. It then ranks every node by relevance to border "background" queries with the closed form $f^* = (D-\alpha W)^{-1}y$. Here $W$ is the region-affinity matrix, $D$ its diagonal degree matrix, $\alpha\in(0,1)$ a smoothing weight, and $y$ the query indicator. The whole computation takes about 0.25 s per image. Saliency Detection via Absorbing Markov Chain instead scores a region by how long a random walk takes to be absorbed at the border, in roughly 0.1 s. Both papers show that diffusion fills the interiors of uniform objects, where purely local contrast leaves "holes."

That interior filling is exactly the property a seam needs. A seam that follows image edges, by contrast, would happily cross the flat inside of an object. Two follow-up lines of work supply method components. Robust Salient Object Detection and Segmentation selects background seeds adaptively when objects touch the border. Salient object detection on hyperspectral images using features learned from unsupervised segmentation task shows that learned feature affinities outperform hand-built color affinities. We reuse both ideas, but we never output a saliency map. Every one of these papers stops at salient-object masks, and none touches vision-language reasoning.

The complexity literature motivates the work and constrains its design. Three papers establish the collapse and rule out the cheap fixes. VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes evaluates 37 models on dense 4K paintings, finds that resolution and scale do not help, and reports o3 at 19.6% on the hardest split. ChartMuseum finds that extended thinking adds only 1.4% on visual chart reasoning. Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs introduces VISER: three labeled horizontal lines plus a prompt to scan sequentially. VISER raises GPT-4o 2D counting from 12.0% to 38.8%, but it degrades at 50+ objects because the lines sever targets, and its authors call for adaptive partitioning.

LVLM-COUNT avoids cutting objects in a different way. It detects objects with GroundingDINO, segments them with SAM, routes A* cuts through background pixels, and counts each crop separately. This drops Emoji-Count error for Qwen2-72B from 78.05 to 24.43. Our proposal differs from VISER in being content-adaptive. It differs from LVLM-Count in three ways: it needs no detector, segmenter or category name; it keeps a single image with a scaffold drawn on it rather than independent crops; and it applies to general question answering. Adaptive-VoCo shows that image complexity can be estimated cheaply to set a global token budget. We borrow only that idea, using it to choose how many seams to draw.

Method

We partition the image into superpixels and connect neighbors. Edge weights $W_{ij}$ are Gaussian in frozen self-supervised patch features, pooled per superpixel. We then compute the diffusion kernel $K=(D-\alpha W)^{-1}$, normalized row-wise. Entry $K_{ij}$ measures how strongly regions $i$ and $j$ share a manifold neighborhood. Unlike local affinity, it stays high across the uniform or textured interior of a single object.

A seam is a left-to-right path along superpixel boundaries. Its cost sums, over every boundary it travels along, the price of separating the two regions on either side:

$$C(\text{seam})=\sum_{(i,j)\in \text{seam}} \Big(\tilde K_{ij} + \lambda\, s_i s_j\Big)$$

Here $\tilde K_{ij}$ is the symmetrized normalized kernel entry, $s_i$ is an optional propagated saliency score, and $\lambda$ weighs the two terms. Running $\lambda=0$ against $\lambda>0$ ablates the boundary-prior saliency term. We find the path with Dijkstra's algorithm.

We draw $n$ non-crossing seams, each constrained to a vertical band, and label them as VISER does. The same sequential-scan prompt is reused unchanged, so the only thing that differs from VISER is where the lines go. We choose $n$ so that each strip holds a bounded number of distinct diffusion clusters, counted by spectral clustering on $K$. A fixed $n=3$ is kept as an ablation. The whole pipeline is a matrix solve on a few hundred nodes plus shortest-path searches, and it should land in the sub-second range the classical papers report.

Research Program

The work has three parts.

First, we build the seam generator and measure fragmentation directly. On the VISER synthetic scenes, object masks are known exactly. On FSC-147, object points are annotated, and we count a point as cut when a seam passes within a fraction of the local object spacing. Fragmentation rate becomes our mechanistic variable.

Second, we test the claim that fragmentation explains the remaining scaffold failures. We regress per-image accuracy gain on fragmentation rate across all scaffold variants. If the claim holds, scaffolds of different origin should fall on a single curve.

Third, we stress the method where the boundary prior should fail: VisualOverload paintings and crowded counting images whose objects fill the frame. The prediction is that the pairwise-kernel seam keeps its advantage while the boundary-prior saliency version collapses toward VISER's performance.

Baselines

The naive reference is the unmodified image with a standard prompt, run on the same models, decoding settings and question sets. Its published numbers anchor the gains: GPT-4o at 12.0% and Qwen2.5-VL at 5.8% on VISER 2D counting; GPT-4o at 25.57 MAE on FSC-147; Qwen2-72B at 78.05 MAE on Emoji-Count.

The most direct competitor is VISER with identical prompt and line count. It isolates the single change we make, line placement. It reports GPT-4o 2D counting of 38.8% and Qwen2.5-VL of 40.8% in Visual Structures Help Visual Reasoning. Beating it means higher accuracy at matched $n$, with the margin growing with object count. We also give VISER a tuning budget equal to ours over $n$ and line offsets, so it is not handicapped.

The strongest method that avoids cutting objects is LVLM-Count. Its reported results are FSC-147 MAE 17.86 for GPT-4o and Emoji-Count MAE 24.43 for Qwen2-72B in LVLM-COUNT. We run it with its released components on the same images and models and record wall-clock time. On counting, matching it within its error while running far faster without a category prompt counts as success. On VisualOverload, where it does not directly apply, we build a hybrid as an upper bound: SAM-mask A* seams drawn as a scaffold.

Two ablations isolate the key ingredient. In the edge-energy seam, cost is image gradient magnitude, with no diffusion. In the boundary-prior saliency seam, the cost uses only $s_i$ from two-stage manifold ranking. For VisualOverload and ChartMuseum there is no published scaffold or partitioning result, so the reference points are the no-scaffold numbers reported by those papers. VisualOverload reports o3 at 69.5% overall and 19.6% hard, and Gemini 2.0 Flash at 41.7% best counting. ChartMuseum reports Qwen2.5-VL-72B at 38.5% and Gemini-2.5-Pro at 63.0%.

Why not just…?

Why not just add more straight lines, or a finer grid? More lines shrink each strip, but they also cut more objects. LVLM-Count's grid ablation shows double counting rising, and VISER itself degrades past 50 objects. We include grids of 2 to 8 lines on the fragmentation-versus-gain curve. If a fine straight grid matched our seams at equal fragmentation, adaptive placement would add nothing, and we would report that.

Why not just route seams along image edges, as in seam carving? This is the sharpest threat, because it is cheaper still. The hole problem predicts that edge energy is low inside uniform objects, so edge seams will pass through them. The edge-energy ablation tests this directly. If it matches the diffusion seam, the graph-propagation component is unnecessary and the contribution shrinks to "content-aware lines."

Why not just use SAM masks? SAM masks plausibly give lower fragmentation than our seams. The question is whether that extra accuracy is worth a multi-model pipeline and a category prompt. The SAM-seam upper bound measures the gap. If the gap is large on natural images, we would recast our method as the fast approximation and report the trade-off honestly.

Why not just use the model's own attention as the saliency signal? Attention becomes diffuse exactly in overloaded scenes, as Multi-Modal Interpretability for Enhanced Localization in Vision-Language Models argues. We nonetheless run an attention-map seam on Qwen2.5-VL as an extra baseline.

Proposed Experiments

The models are Qwen2.5-VL (open, for attention access) and GPT-4o, plus one reasoning model on VisualOverload. Benchmarks are the VISER binding generator, binned by object count; FSC-147 and Emoji-Count, measured by MAE; VisualOverload, by per-task and hard-split accuracy; and ChartMuseum, by accuracy on visual-reasoning questions. ChartMuseum is a stress test, since chart elements are thin and seams may interfere with reading them.

The first decisive check is density-binned accuracy. Our seams should beat VISER by a margin that grows with object count and is near zero for sparse scenes.

The second is the fragmentation curve. If accuracy gain is not explained by fragmentation across scaffold types, the core claim fails.

The third compares the kernel seam with the boundary-prior seam on VisualOverload. We predict the boundary-prior version loses most of its advantage there, which would confirm that pairwise diffusion affinity, not saliency, is the useful signal.

The fourth is latency against LVLM-Count. The file reports no LVLM-Count runtime, so we measure it ourselves.