Is the Dense-Scene Performance Cliff a Perception Failure or a Prompt Artifact?

Vision-language models (systems that answer questions about images in text) do well on sparse photos and fail sharply on crowded ones: shelves of products, dense charts, paintings full of small figures. Several recent benchmarks document this "performance cliff" by growing the number of objects or chart marks and watching accuracy fall. Each of them, however, asks each question with a single fixed wording. A separate body of work shows that the same models can change their answers when an instruction is lightly reworded or the multiple-choice options are reordered. This brittleness tends to be worst exactly where the model is least confident. If that link holds for images, then part of what dense-scene benchmarks report as "the model cannot see" may instead be "the model's answer depends on how you asked." In that case the model rankings on those benchmarks are least trustworthy in precisely the regime they were built to measure.

This proposal measures prompt sensitivity as a controlled function of visual density. It uses the answer's dependence on wording to split each model's errors into two kinds: errors that persist under every reasonable phrasing, which point to perception, and errors that come and go with phrasing, which point to instruction handling or low-confidence decoding. For open models, internal probes of the image encoder give an independent check on which kind of error is which. The output is a density-indexed evaluation protocol. It reports how much of each model's accuracy, and each leaderboard ordering, survives a change of wording.

Core Claim

The claim is this: as scene density grows, the fraction of items whose correctness flips across semantically equivalent prompts rises through the cliff region and then falls again at extreme density. At extreme density, errors become prompt-invariant because the visual information never reaches the language model. Near the cliff, rank correlations between models computed under different prompts drop well below those seen on sparse scenes. Prompt-dependent errors should also concentrate on items where the needed information is still linearly decodable from the vision encoder.

Each part of this is testable and could fail. Sensitivity might instead be flat in density, or rise monotonically, or be unrelated to encoder decodability. Any of those outcomes would still tell benchmark builders whether single-prompt dense benchmarks can be trusted.

The claim builds on three pieces of prior work. ProSA found in text-only models that sensitivity tracks flat output distributions and grows with task difficulty. Tartaglini et al. showed on the FUGU scatter plots that point coordinates are nearly perfectly decodable in the vision encoder but lost after the handoff to the language model. That makes "information present but badly used" a real category of error, distinct from "information absent." Shen et al. showed that simply permuting options can drive vision-language models below chance: LLaVA-7B on ScienceQA falls from 45.20% to 2.28%. They tested only ordinary, uncluttered images.

How This Differs From the Closest Work

The nearest existing papers each cover two of the three pieces. The dense-scene benchmarks vary density but hold wording fixed: VisualOverload (4K paintings, 37 models), ChartMuseum (expert chart questions with mark-density sweeps), FUGU, and VLM-SubtleBench (object counts from 4 to 32). VisualOverload's logical-consistency pairs ("is it day?" and "is it night?") probe answer coherence, not wording invariance.

The prompt-sensitivity and leaderboard studies vary wording but not density. Alzahrani et al. and ProSA are text-only. Shen et al. use ordinary images. VHELM standardizes one template per scenario and notes that open models often fail on formatting rather than seeing. Promise reduces template variance, but only for CLIP-style classification.

VISER is the one paper that couples density with instructions. It pairs drawn lines with one specific scanning command and never measures sensitivity to that command.

This proposal differs from all of them in three ways. It sweeps both factors jointly under controlled generation. It turns cross-prompt agreement into an error decomposition. It validates that decomposition against encoder probes rather than treating variance as a nuisance to average away.

Research Program

Density-controlled image sets. The first component is a density-controlled image set with a matched prompt family. On the synthetic side, the FUGU procedure is regenerated with densities extended from the original $n \le 16$ to $n = 64$ points, so the full cliff and the region past it are covered. The VLM-SubtleBench existence-style clutter test is rebuilt from 4 to 64 objects. On the natural side, ChartMuseum's mark-density variation is used, and VisualOverload items are binned by an annotated or automatically estimated entity count.

Prompt variants. Every item gets $K = 16$ prompt variants. These cover the four ProSA template categories (plain, supportive framing, role play, explicit output requirement), paraphrases of the question checked by two annotators for equivalence, and, for multiple-choice items, a fixed random sample of option permutations and label schemes (letters versus Roman numerals, following Shen et al.).

Answer extraction. Answers are extracted with a lenient parser as well as a strict one. This keeps formatting failures of the kind VHELM documented separate from content changes, and both rates are reported.

Error decomposition. For item $i$ at density $d$, call the item robustly correct if it is correct under all $K$ prompts, robustly wrong if wrong under all, and prompt-dependent otherwise. The central quantity is the prompt-dependent share $S(d)$, the fraction of items at density $d$ that are prompt-dependent. It is tracked alongside the robust-correct and robust-wrong curves. $S(d)$ is also compared against the value it would take if each prompt were an independent coin flip with the model's per-density accuracy. That comparison says whether wording effects are structured or merely noise.

Leaderboard stability. For each density bin, Kendall's $\tau$ between model rankings is computed under every pair of prompts. This is Alzahrani et al.'s rank-stability measure, moved to images and indexed by density.

Encoder probes. For open models (InternVL3, LLaVA-OneVision, LLaMA-3.2-Vision, Qwen2.5-VL), linear probes on vision-encoder layers are trained following Tartaglini et al., to decode the answer-relevant quantity: count, coordinate, or presence. This labels each item as "information decodable in encoder" or not. The decomposition is validated only if prompt-dependent items are enriched among encoder-decodable ones and robustly-wrong items are enriched among non-decodable ones.

Scope and budget. GPT-4o is included for the behavioral analyses, without probes. Every model uses greedy decoding, identical image resolution, and identical token budgets, as in VHELM.

Baselines

Single fixed prompt (naive reference). This is what VisualOverload, ChartMuseum and VHELM report: one standardized template, greedy decoding. It is run with the benchmarks' own templates on the same items and models. For scale, VisualOverload reports 69.5% overall and 19.6% on its hardest split for o3. ChartMuseum reports 63.0% for Gemini-2.5-Pro. FUGU reports 69.3% average for InternVL3. The proposal beats this baseline if the single-prompt accuracy at cliff densities falls outside the range spanned by the $K$ variants often enough to change conclusions. Concretely, that means at least one pair of models whose ordering under the benchmark template reverses under a majority of variants.

Permutation robustness from Shen et al. This is the strongest existing method for multiple-choice robustness in vision-language models: evaluating across option permutations, with majority vote as the mitigation. It is run with the same sampled permutations and compute. Its reported effect is the LLaVA-7B ScienceQA collapse from 45.20% to 2.28% under adversarial permutation. The protocol goes beyond it if paraphrase and framing variants identify prompt-dependent items that permutations alone miss, measured as the fraction of prompt-dependent items with no permutation-induced flip.

Output entropy and per-item prompt sensitivity. These are the confidence-based signals from ProSA and SENSEL, which uses stability under rewording as an abstention score. SENSEL reports +4.1 AUC on the F1-coverage curve over max-probability for GPT-J-6B. No reported result was found on Lacuna for PromptSensiScore on vision-language models or dense scenes. All signals are computed from the same forward passes, so compute is matched. The decomposition counts as adding value only if prompt-dependence predicts encoder decodability better than output entropy does, compared by area under the curve on held-out items.

Ablation. One ablation isolates the key ingredient, the joint density sweep. The same $K$-prompt analysis is run on density-shuffled item pools, with density ignored. If aggregate sensitivity looks the same and density adds no explanatory power in a regression of flip rate on density plus item difficulty, then the density-indexed claim fails.

Why not just…?

Why not just run existing dense benchmarks with a few paraphrases and report error bars? This is the real threat. If sensitivity turns out to be flat in density, it is essentially what this proposal would recommend anyway. The difference is that uncontrolled benchmarks mix density with question type and image source, so any trend there is uninterpretable. Error bars alone also cannot say which errors are perceptual. The synthetic sweep and the probe validation are what the simpler version lacks. If the density ablation above shows no density effect, the proposal collapses to this simpler recommendation, and that negative result is itself worth reporting.

Why not just use the model's output entropy as the confidence signal? ProSA ties sensitivity to flat output distributions, so entropy might carry the same information at one-sixteenth the cost. Two pieces of evidence argue against this. Rethinking Hallucinations found that entropy-style detectors track consistency rather than correctness. Entropy is also defined per prompt, whereas the question here concerns stability across prompts. The entropy baseline comparison against encoder decodability settles this directly. If entropy matches prompt-dependence, the cheaper signal should be adopted.

Why not just fix the brittleness with permutation majority vote or prompt-robust fine-tuning? Mitigation is not the goal. Voting across 24 permutations multiplies cost and, per Shen et al., does not restore accuracy. Promise covers only classification with contrastive image-text models. A vote would also hide exactly the distinction this work needs, between errors that wording can reach and errors it cannot. Majority-vote accuracy per density is reported as a secondary baseline, to show how much of the cliff voting recovers.

Why not just attribute everything to item sampling noise? The prompt-induced variance is compared with bootstrap variance over items within each density bin. Rank instability is attributed to prompts only when prompt-to-prompt $\tau$ falls below what item resampling alone produces.

Decisive Checks

The first decisive result is the shape of $S(d)$ on the extended FUGU and clutter sweeps, compared against the independent coin-flip reference. A peak near the density where robust-correct accuracy halves would support the core claim. A flat curve would refute it.

The second is the probe test. Among open models, if prompt-dependent items are no more encoder-decodable than robustly-wrong ones, the perceptual-versus-instructional reading of the decomposition is wrong, and the work reduces to a variance audit.

The third is transfer to natural images. On ChartMuseum and VisualOverload bins, prompt-to-prompt Kendall's $\tau$ is checked for a fall with density toward or below the roughly 0.5 that Alzahrani et al. saw for text leaderboards. If it does, dense-scene leaderboards need multi-prompt reporting. If sparse and dense bins look equally stable, current single-prompt practice is vindicated for these benchmarks.