When a vision-language model is asked to count the cars in a parking lot or the blueberries on a bush, a wrong answer can come from three different sources. The model may never have resolved the crowded, overlapping objects in the first place. It may be guessing from the word in the question, for example "a carton of eggs is twelve." Or it may be recalling the answer because the benchmark image leaked into its training data. Today's counting benchmarks report one error number that mixes all three. This proposal builds a test that pulls them apart. We edit benchmark images in ways whose new count is known exactly from the existing point annotations, such as crops, mosaics, and pasting annotated instances back into the image. We cross these edits with questions that use the real object name, a nonsense name, or a misleading name. We then sort every image by how much its objects overlap. A model that recalls a leaked answer will repeat the old count on an edited image. A model that leans on the name will change its answer when the name changes and the pixels do not. A model that truly fails on overlap will fail in proportion to overlap whatever the name or edit.
The idea that matters is that none of these edits needs an image generator. The closest prior work, Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations, detects leaked test sets by redrawing images with a diffusion model so that a different multiple-choice answer becomes correct. That paper notes that its generator struggles with intricate multi-object geometry, and dense scenes are exactly that case. Point annotations give us something it lacked: exact ground truth after editing, in precisely the crowded scenes where generation is unreliable.
Core Claim
We claim two things. The first is methodological: generation-free, annotation-exact, count-changing edits give a reliable contamination signal for dense counting. The signal is the anchor rate, meaning the fraction of answers on an edited image that equal the original image's ground-truth count when the true count has changed. The second is scientific, and it is the claim we expect to be most interesting: non-visual shortcuts take over as overlap grows. In high-overlap images, a model's answer should shift more under name changes, and a contaminated model should anchor more to memorized counts, than in sparse images of the same benchmark. This would extend to dense counting what Multi-Object Hallucination in Vision Language Models observed with separated, boxed objects: when models hallucinate, less than 20% of their attention goes to visual tokens. It would also extend what WS-COC reported about severe undercounting in dense scenes. If the interaction is absent, so that name sensitivity and anchoring are flat across overlap strata, the scientific claim fails, though the contamination tool may still be useful.
Research Program
The edit families are built from point or box annotations in FSC-147, CARPK, ShanghaiTech, DIOCblueberry, and the Locount dataset. Crops select a sub-window and count the annotated points inside it. To avoid the tile-boundary splitting that MaskCount identified as a major error source, we place crop edges along low-density paths in the density map. We also report a tolerance band equal to the number of instances cut by the boundary. Mosaics tile two to four test images, so the true count is the sum. A memorizing model should output one constituent's count or a stale value. Instance pastes copy annotated instance patches from the same image into its dense regions at a controlled box overlap (intersection-over-union) with existing instances. This is the only edit that raises overlap on purpose. Because pasting can leave visible artifacts, we treat it as the overlap manipulation rather than the contamination probe, and we check it with human spot counts.
The question conditions borrow the logic of LLMs Lean on Priors, Not Programming Language Semantics. That paper found that models handle alien symbols better than familiar symbols given new meanings, with drops of 40 to 60 points under the swapped condition. In our version, the true-name condition asks "count the eggs." The nonsense-name condition asks "count the blickets," supplies one exemplar box drawn on the image, and states that a blicket is the boxed object. The swapped-name condition gives the name of a different FSC-147 class along with the same exemplar box and states that this word refers to the boxed object. The instruction is identical in the nonsense and swapped conditions, so the only difference is whether the word carries a conflicting prior. Overlap strata come from annotations: mean pairwise box overlap among nearest neighbors where boxes exist, and local point density with nearest-neighbor distance normalized by exemplar size elsewhere.
Every model is scored on the full grid of edit, question condition, and overlap stratum. We report mean absolute error (MAE), the anchor rate, and the name-shift. The name-shift is the change in a model's answer between the nonsense and swapped conditions on the same pixels. We calibrate the anchor rate by building known-contaminated models. Following the protocol of Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations, we fine-tune LLaVA-OneVision-7B on FSC-147 test images paired with their counts, for one and three epochs, and at leak rates of 100% and 10% of the test set. This gives a dose-response curve that an anchor rate measured on any unknown model can be read against.
Baselines
The naive reference is the plain static benchmark score: MAE on unedited FSC-147 test images with true class names. This is what papers report today. The strongest existing counting system in the file that answers through a language interface is WS-COC. It fine-tunes LLaVA-OneVision-7B with dialogue tuning, count ranking, and averaging of global and local 2×2 counts, and it reports FSC-147 test MAE falling from 38.19 for the zero-shot model to 13.91, and from 149.69 to 54.37 on scenes with more than 100 objects. We will run the released WS-COC model and its zero-shot backbone on our grid, using the same images and prompts and no extra tuning. We do not aim to beat WS-COC's MAE. We want to show how much of its 13.91 survives edits and name changes. A clear result would be a gap in MAE between original and crop images that remains after accounting for the smaller counts in crops, or a name-shift that rises with overlap.
The strongest existing contamination detector is the diffusion-perturbation method of Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations. On MMStar it reports a score delta of +31.5% for clean models and as low as -45% for contaminated ones. For LLaVA-v1.5-7B, the clean model moved from about 38% to about 69% and the 3-epoch contaminated model moved from about 54% to about 38%. We will apply its Flux plus ControlNet pipeline to the same FSC-147 images, prompting for edits such as "add three more objects." We give it the same number of edited images per original as our method and the same filtering budget, and we score both methods on how well they separate our calibrated contaminated models from clean ones. We beat it if our anchor-rate detector separates the 10%-leak model from the clean model at a lower false-positive rate. We also report how often its generated images have a human-counted total different from the intended one.
Non-language counters provide a reference that cannot have name priors. These are MaskCount and the LOCA and CounTR models it compares against; MaskCount reports a 49.16% MAE reduction over them on DIOCblueberry. Their error per overlap stratum gives the "pure perception" curve that language-model curves are compared against. Two ablations isolate the key ingredient. The first replaces our count-changing edits with count-preserving ones such as flips and color jitter, under which a memorized answer stays correct, so the anchor rate should carry no signal. The second removes overlap stratification and reports a single pooled score, to test whether the interaction claim depends on stratifying.
Why not just…?
Why not just evaluate on a newer dataset the model cannot have seen? New datasets like DIOCblueberry are valuable and we include them, but a drop on a new dataset mixes contamination with a genuine change in the kind of images. Blueberries under foliage differ from FSC-147 in object scale, by a factor of 7.9 in box size, as well as in exposure. Our edits keep the source distribution almost fixed and change only the answer, which is the property that let LINGOLY-TOO attribute its score drop to removed shortcuts rather than to new material. New datasets also leak in turn, while edits can be regenerated.
Why not just reuse the diffusion perturbation method? Its own authors flag dense, fine-grained geometry as a weakness, and dense scenes are where counting is hard. Whether that weakness is fatal is an empirical question that our baseline comparison measures directly through human recounts of generated images. If diffusion edits turn out to be count-accurate on FSC-147, the remaining argument for our method rests only on cost and exactness, and we will say so.
Why not just use random crops as a contamination check and skip the rest? This is the most serious alternative. Crop-based anchor rates may on their own carry most of the contamination signal. If the calibrated experiments show that, the contamination part of this work reduces to a simple recipe, and that is a useful finding in itself. The name conditions and overlap strata address a different question that crops cannot answer: whether errors come from priors attached to words or from merged visual boundaries. The ablation that drops stratification tests directly whether this extra structure adds information.
Why not just ask the model whether it has seen the image, or check text overlap? Lexical overlap checks cannot see visual memorization; INVITE and the VLM perturbation paper both note that recall can survive rephrasing. A count, however, is a single number with no wording to rephrase, so a check based on behavior under edits is the more direct test.
Proposed Experiments
The decisive checks are three. First, on the calibrated LLaVA-OneVision models, the anchor rate on crops and mosaics must rise with leak rate and epochs and stay near the chance level of exact matches for the clean model. If the 10%-leak model is indistinguishable from the clean one, the detector is too weak to be worth deploying. Second, the overlap interaction: name-shift between the nonsense and swapped conditions, and anchor rate for contaminated models, should increase across overlap strata. Instance pastes provide a causal version of the test, because they raise overlap within a single image while holding identity and name fixed. The non-language counters should show rising error but, by construction, no name-shift. Third, we apply the grid to off-the-shelf models, including the WS-COC model, its zero-shot backbone, and proprietary models such as GPT-4o. This reports how much of each model's published counting error is attributable to overlap, to names, or to anchoring. There is no reported result on Lacuna for counting with obfuscated or swapped class names, so the name-shift numbers would be new.
References
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- Multi-Object Hallucination in Vision Language Models
- Bootstrapping MLLM for Weakly‑Supervised Class‑Agnostic Object Counting
- Benchmarking Dense and Indiscernible Object Counting with Blueberries
- LLMs Lean on Priors, Not Programming Language Semantics
- LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations