Vision-language models fail sharply once a scene holds more than a handful of objects. They miscount, they assign the wrong color to the wrong shape, and they lose track of points in crowded charts. We do not know where inside the system this failure happens. One possibility is that the image encoder never records the crowded objects. Another is that it records each object correctly at its own location, but the information is lost when those local features are pooled into a summary or passed to the language model. These two explanations call for different fixes: a better encoder in the first case, a better readout in the second. Current diagnostics cannot tell them apart.
We propose a measurement that can. A linear probe is a single trainable linear layer placed on top of a frozen network, used to test what information that network's features already contain. We freeze the encoder and attach the probe as a $1 \times 1$ convolution, so it reads every spatial location of an intermediate feature map rather than one pooled vector. We then sweep scene density in a controlled way. To test causes, we also intervene: we multiply the frozen feature map by a spatial attention mask that keeps the target objects and suppresses the rest, and we check which kind of probe recovers. Comparing three quantities as density grows (local decodability, pooled decodability, and recovery under masking) separates "the features are gone" from "the features are there but get mixed together on the way out."
Core Claim
Our hypothesis is that in transformer-based encoders, single-object attributes and even attribute pairings remain linearly decodable at each object's own location well past the density where pooled probes and the full model's answers collapse. If so, much of the "performance cliff" is a failure of readout and of the vision-to-language handoff, not of perception.
We predict two further patterns. Masking the frozen features with an object mask before pooling should recover most of the pooled-probe loss. Masking at an early layer and letting the remaining frozen layers run should help transformers more than convolutional networks, because transformers mix information globally at every layer, while convolutions only mix within a local neighborhood.
The hypothesis is falsifiable. If the local probes collapse at the same density as the pooled probes, the encoder itself is the bottleneck, and readout-side fixes such as attention pooling or visual prompting are aimed at the wrong place.
Relation to Prior Work
Three lines of work motivate the question.
-
VisualOverload evaluates 37 vision-language models on dense 4K paintings. The best counting accuracy is 41.7%. The authors blame token compression in the encoder, but they never inspect the features themselves. - Diagnosing Bottlenecks in Data Visualization Understanding reports that point localization and counting collapse once density exceeds a few items, and frames this as a vision-language handoff problem. It reports no number the file gives us.
-
Visual Structures Help Visual Reasoning attributes the collapse to the binding problem, meaning attributes get attached to the wrong objects. Its fix, VISER, draws horizontal lines on the input image and raises Qwen2.5-VL synthetic counting from 5.8% to 40.8%. Because the fix acts on pixels, it cannot say which internal stage it repairs.
Our proposal supplies the internal measurement these papers lack, and it moves the intervention from pixels to features.
Two papers supply the methods we reuse.
- Deep Gaze Pooling multiplies frozen CNN feature maps by a fixation density map, then pools and classifies without retraining. That is exactly our masking operation $F \odot M$. However, it is applied to collages of 20 well-separated images and is used to decode a user's intent, not as a diagnostic across density.
- Benchmarking LVLMs on Fine-Grained Image Tasks uses linear probes on frozen backbones to expose an "alignment tax." Its probes use only global embeddings of centered single objects. We keep the probing logic but make it spatial and indexed by density.
Two papers constrain the design.
- Human Attention in Fine-grained Classification finds that Grad-CAM masks are much weaker than human gaze: keeping the top 5% of pixels gives 81% accuracy with gaze versus 70% with Grad-CAM. Any mask predicted by the machine therefore has to be compared against an oracle mask.
- Instance-Sensitive Fully Convolutional Networks warns that per-location convolutional readouts cannot separate touching instances of the same class. For this reason our dense probes predict attributes and density, which do not require instance identity, rather than instance labels.
Per-location linear heads are already common for evaluating segmentation, so the probe architecture alone is not new. What is new is using it, together with masking interventions, to locate the stage at which dense scenes break.
Research Program
Synthetic scenes. We build a controlled generator in the style of the VISER synthetic scenes, with 2D and simple 3D variants. Object count $N$ sweeps from 1 to over 100. Object size and spacing are varied independently, so that pure resolution loss can be told apart from crowding. Each object has a color and a shape, and distractors share single attributes with targets, which is the condition that stresses binding.
Encoders. We test frozen encoders spanning both architecture families:
- a ResNet-50, which is fully convolutional;
- CLIP and DINOv2 vision transformers, chosen because the fine-grained benchmark found these contrastive models retain the most separable features;
- the vision towers of open vision-language models such as Qwen2.5-VL.
Four probe families. All four are trained on the same frozen features at each layer.
- Pooled probe. A global linear probe on the averaged or CLS embedding (the CLS token is a summary vector a transformer produces for the whole image). It predicts the count and whether a given color-shape pairing is present.
- Local attribute probe. A $1 \times 1$ convolutional probe that predicts the color and shape of whatever object covers each location.
- Local binding probe. A $1 \times 1$ probe that predicts the joint color-shape class at each location. A model can succeed on the separate attributes yet fail here, which is how binding failure shows up.
- Density probe. A linear density-map probe whose count is $$\hat{c} = \sum_{h,w} \left( w^\top F_{:,h,w} + b \right),$$ where $F_{:,h,w}$ is the frozen feature vector at location $(h,w)$, and $w$, $b$ are the only trained parameters.
Locating the break point. For each failure mode we define a critical density $N^*$: the count at which a probe's accuracy falls to halfway between its accuracy at $N=1$ and chance. The central quantity is the gap between the local and pooled $N^*$ values. A large gap means the encoder keeps the information but pooling loses it.
Interventions. We use three masks $M$:
- an oracle mask built from ground-truth object masks of the queried objects;
- a predicted mask from a saliency model;
- a Grad-CAM mask.
Each mask is applied in two ways. First, at the readout: multiply the final features by $M$ before pooling. Second, mid-network: multiply the features at layer $l$ by $M$ and let the frozen layers after $l$ run. Recovery from readout masking tests whether pooling interference explains the failure. Recovery from mid-network masking tests whether distractors corrupt the target's features during later processing. We expect transformers and convolutional networks to differ on this second test.
Baselines
Naive reference. The pooled linear probe with no mask. This is the standard protocol from the fine-grained benchmark. It sets the floor: its $N^*$ is the "cliff" as conventional probing would report it.
Strongest learned readout. A probe that keeps the backbone frozen but pools with a learned attention mechanism (a few trainable queries that attend over all locations). It gets the same training data, optimizer, and hyperparameter search budget as our probes. This is the most serious competitor, because a learned soft mask applied at readout might do everything our oracle masking does. Beating it means the analysis shows something attention pooling cannot: specifically, a case where mid-network masking recovers accuracy that readout pooling of any kind cannot reach.
Strongest trained dense counter. On real crowd counting, the end-to-end trained fully convolutional counter with scale alignment from Towards A Universal Model for Cross-Dataset Crowd Counting. That paper reports UCF-QNRF mean absolute error (MAE) falling from 88.7 for its baseline to 79.2 with alignment. We do not expect a frozen linear probe to beat 79.2. The meaningful result is how much of the distance from naive to fully trained the linear readout covers, and whether that fraction shrinks with density faster for some backbones than others. A frozen probe that lands near 88.7 would show that the frozen features already carry nearly all the counting information an untrained-alignment baseline uses. MPCount, which gates density maps using predicted patch-level head presence, serves as a reference for mask-based gating under domain shift. We cite its relative error reductions rather than absolute values, because the file gives only relative figures.
Pixel-space intervention. For vision-language models, VISER line scaffolds, run on the same synthetic scenes so that input-space and feature-space interventions can be compared at equal density.
Mask-content ablation. Masks with the same area as the oracle but placed randomly, and a uniform mask. These isolate the key ingredient: they test whether the recovery comes from where the mask points or merely from shrinking the pooled region.
Why not just…?
Why not just use a learned attention-pooling probe and skip the masking? This is a real threat to the readout-side half of the claim. If learned pooling matches oracle readout masking at every density, then the readout masking adds nothing beyond a sanity check. We will report exactly that if it happens. Even so, learned pooling cannot test the mid-network question, because it never changes what later layers compute. The interventional result therefore stands or falls on its own evidence.
Why not just tile the image into high-resolution crops? Tiling confounds resolution with crowding, because each crop holds fewer objects. VisualOverload also observes that encoders which split the image into tiles still fail, partly because tile edges cut through objects. Our generator varies object size and count independently. We will run tiling as a control: if tiling restores the local $N^*$ but not the pooled $N^*$, that again points to readout rather than resolution as the problem.
Why not just fine-tune end to end? Fine-tuning may well perform better, as the crowd-counting results show. But a fine-tuned network cannot tell us whether the pretrained features held the information, which is the question that decides between redesigning encoders and redesigning readouts. Including the fully trained counter as an upper bound puts the size of that trade-off on the table instead of hiding it.
Why not just prompt the model to reason step by step, or use a larger model? Visual Structures Help Visual Reasoning reports that text-only reasoning prompts fail when the visual features are misbound. VisualOverload reports that accuracy sometimes falls as model size grows within a family. We will include two model sizes per family so that our critical densities can confirm or contradict the scaling finding.
Proposed Experiments
The experiments proceed in three stages, ending with a decisive check.
Stage 1: synthetic sweeps. On the synthetic scenes, we plot all four probe types against $N$ for every encoder and layer. The primary result is the gap between the local and pooled critical densities.
Stage 2: real images.
- Counting. We train frozen linear density probes on UCF-QNRF and ShanghaiTech, and test zero-shot on UCF_CC_50. The 88.7 and 79.2 MAE results above are the reference points. The UCF_CC_50 figures from the same paper, 309.6 falling to 244.7, are the zero-shot references.
- Gaze masks. On CUB-GHA, we multiply frozen features by human gaze maps and by predicted gaze maps. This tests whether the 81% versus 70% gap between gaze and Grad-CAM carries over from keep-and-retrain to frozen-feature masking.
Stage 3: the decisive check. For Qwen2.5-VL, we compare the probe-based critical density of its own vision tower with the model's answer accuracy on the same synthetic counting scenes, where its reported baseline is 5.8%. We also run the counting split of VisualOverload on that model. That benchmark has no dense labels, so it can only show the gap between the model's answers and the ceiling of a local probe trained on synthetic data. If the vision tower's local probes still count well at densities where the model answers near chance, the bottleneck is in the vision-to-language handoff. In that case the field should stop scaling encoders for dense scenes and start redesigning how their spatial features reach the language model. If instead the local probes collapse at the same density as the answers, the encoder is to blame, and the masking results tell us whether early suppression of distractors is a viable fix.
References
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
- Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models
- Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs
- Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
- Human Attention in Fine-grained Classification
- Instance-Sensitive Fully Convolutional Networks
- Towards A Universal Model for Cross-Dataset Crowd Counting
- Single Domain Generalization for Crowd Counting