Multimodal language models now score above 90% on benchmarks that ask about tiny objects in very large images. CVSearch reports 91.6% on V*Bench with LLaVA-OV-7B. These results are taken as evidence that "visual search," meaning zooming into high-resolution crops before answering, has fixed the blur that comes from shrinking a 2000-pixel image to a few hundred pixels. A correct answer can arise in three other ways, though. The search may find the right crop and read it. The model may guess from language priors ("the umbrella is probably red"). Or it may recall the answer because the benchmark image, which for V*Bench comes from the public SA-1B collection, appeared in its training data. Accuracy cannot tell these apart, and every contamination test in the literature was built for low-resolution, multiple-choice benchmarks.
We propose a simple counterfactual. Break each answer into small single-fact claims whose supporting image region is known. Then degrade or erase only that region, leaving the rest of the image at full resolution. If the model is perceiving, its support for a claim should fall as the region blurs and should collapse when the region is removed. If the claim survives erasure, the model never needed to see the evidence. Because the rest of the image is untouched, a model that has memorized the image is still able to recall its answer, so erasure also exposes image-conditioned memorization, which text-only "blind" tests cannot detect. Logging which crops a search pipeline actually inspected adds a third measurement: the region-proposal failures behind each error.
Core Claim
Reported gains from high-resolution search mix three quantities: whether the system proposed a crop containing the evidence, whether it read the evidence correctly once proposed, and how often it answers correctly without ever having the evidence. Only the first two are perception. We claim that a per-claim evidence-erasure curve, which measures support for a claim as the target region's resolution goes from native down to zero, separates these quantities on existing static benchmarks. We further claim that this separation predicts how much of a model's score on a static benchmark carries over to freshly photographed, never-published images, and predicts it better than existing contamination detectors.
Concretely, let $h$ be the event that a pipeline's proposed crops overlap the annotated target region $R$. A pipeline's accuracy $A$ then splits as
$$A = P(h)\,a_{\text{hit}} + \big(1-P(h)\big)\,a_{\text{miss}},$$
where $a_{\text{hit}}$ is accuracy when the evidence was inspected and $a_{\text{miss}}$ is accuracy when it was not. The term $a_{\text{miss}}$ above chance is the signature of prior or memorized answering. For a single atomic claim $c$, we define its evidence dependence as
$$D(c) = p(c \mid s_{\text{native}}) - p(c \mid s = 0),$$
where $p(c \mid s)$ is the model's probability of affirming $c$ when $R$ is rendered at effective resolution $s$, and $s=0$ means $R$ has been inpainted away. A perceiving model has $D$ near its native accuracy minus chance. A memorizing model has $D$ near zero.
How This Relates to Prior Work
Three lines of work motivate the gap. The high-resolution search systems V*, Retrieval-Augmented Perception, and CVSearch zoom into the image by different means: guided by language-model heuristics, by retrieving crops as documents, or by experts with adaptive patching. All three report only end-to-end accuracy on static benchmarks. CVSearch also explains its failures as proposal misses, where an expert such as SAM fails to suggest the small object. None of them measures how often they answer correctly without a hit, which is what our decomposition isolates. The audit treats these systems as subjects, not competitors.
The contamination literature supplies the concern and constrains the design. Both Text and Images Leaked! shows leakage through both text pre-training and instruction tuning. It detects leakage by shuffling options and by slot-guessing on paraphrases, and these perturbations act on the text, not the visual evidence. Vision-Language Bootstrapping edits images by adding distractors, removing non-essential objects, and outpainting. It reports image overlap with pre-training data as high as 84.46% and contamination on MME falling from about 33% to about 10%. It works at standard resolution, though, and must re-verify labels with a model-based judge because edits can change the correct answer. We differ on that point: we remove the essential object on purpose. The ground-truth change is then known by construction (the claim should no longer be affirmable), so no judge is needed. We do reuse that paper's inpainting tool, PowerPaint.
Atomic decomposition supplies the method. Pelican breaks visual claims into predicates such as Exists and Position and verifies them against a detector-built table. That makes its verdicts depend on detector recall, the same proposal ceiling we are measuring. Ask, Don't Judge shows that yes/no questions per atom track human judgment better than holistic scores. We use decomposition as an experimental control, not a verifier. A free-form answer about a scene produces claims tied to the erased target (its color, its text) and claims tied to an untouched anchor object. Erasure should remove the first kind and leave the second intact, so every item carries its own internal control.
Research Program
The first component is an annotated audit set. V*Bench (191 SA-1B images, average 2246×1582) and HR-Bench 4K/8K questions are paired with human-drawn boxes for the target region and for each anchor object the question mentions. Annotation is cheap at this scale. The file does not say whether these benchmarks ship target boxes, so we budget for drawing all of them. For each item we elicit a free-form answer plus the original multiple-choice answer. A fixed decomposer turns the free-form answer into atomic yes/no claims, and each claim is assigned to the target or anchor region by a human check on a sample.
The second component is the region-local degradation ladder. Only $R$ is downsampled by factors of 2, 4, 8, and 16 and then upsampled back, using a feathered mask to avoid visible seams. At $s=0$, $R$ is removed with PowerPaint. A placebo condition applies the identical operation to a same-size region that contains no queried object. This controls for the model reacting to editing artifacts instead of to the missing evidence.
The third component is pipeline instrumentation. For SEAL, RAP, CVSearch, and a Pelican-style detector pipeline, we log every crop or box inspected. This gives $P(h)$, $a_{\text{hit}}$, and $a_{\text{miss}}$ at each rung of the ladder. It also shows whether proposal failure begins at the coarse "glimpse" stage, as Direction 3's analysis of low-resolution glimpses suggests.
The fourth component is the validation set: roughly 300 new high-resolution photographs we take ourselves after the training cutoffs of the audited models, with question templates matched to V*Bench and HR-Bench. On this set, contamination is impossible by construction.
Baselines
The naive reference is raw benchmark accuracy, taken as a measure of perception. The strongest reported system number is 91.6% on V*Bench for LLaVA-OV-7B with CVSearch, alongside 75.39% for SEAL in V*. We will rerun these with released code and report the gap between each system's static and fresh-set accuracy. Our claim is that this gap, not the headline number, measures real perception.
The strongest competing diagnostics are the following. The blind text-only test asks the question with no image. MM-DETECT's option-order sensitivity test, from Both Text and Images Leaked!, produces a perturbation drop Δ. VLB's distractor and removal edits, from Vision-Language Bootstrapping, give an accuracy drop. Each will be run on exactly the same items and models, with the same number of model calls per item where possible. No reported result was found on Lacuna for MM-DETECT or VLB applied to V*Bench or HR-Bench. The only published Δ values are on ScienceQA (−5.3 for Claude-3.5-Sonnet) and Vintage (−5.7 for Phi-3-vision).
Two ablations isolate the key ingredients. Global downsampling of the whole image tests whether local erasure matters. Holistic multiple-choice erasure without decomposition tests whether atom-level anchor controls matter. Beating the baselines means the following: across models and benchmarks, mean evidence dependence $D$ explains significantly more variance in the static-to-fresh accuracy gap than blind accuracy, MM-DETECT Δ, or VLB drop. We test this with rank correlation over at least eight model–pipeline pairs, with bootstrap intervals.
Why not just…?
Why not just ask the question without the image? The blind test measures pure language priors. It cannot detect a model that recognizes a familiar SA-1B image and recalls its answer, because that model needs the image to answer. Erasing only the target keeps the image recognizable, so memorization survives erasure while perception does not. If blind accuracy predicts the fresh-set gap as well as $D$ does, image-level memorization is negligible, and our method reduces to a costlier prior test. That is a publishable negative result, and the comparison above settles it.
Why not just downsample the whole image and watch accuracy fall? Global downsampling degrades the target, the anchors, and the coarse glimpse that search pipelines use to decide where to zoom, all at once. A drop cannot be attributed to any one of these. Region-local degradation keeps the glimpse intact for everything except the target. The global-downsampling ablation shows how much this attribution changes conclusions.
Why not just run MM-DETECT's option shuffling on V*Bench? Shuffling detects memorized answer formats. A model that has memorized "the umbrella is red" answers correctly in any option order. We expect shuffling Δ to be small on high-resolution benchmarks even where $D$ shows non-perceptual answering. If shuffling Δ instead predicts the fresh gap as well as $D$, the cheaper test wins, and we will say so.
Why not just build a fresh benchmark and skip the audit? This is the real threat. A fresh set is the gold standard, and we build one ourselves. It is expensive, though, becomes stale once published and crawled, and gives no per-item explanation of why a system succeeds. The audit's value depends on reproducing the fresh-set verdict cheaply on existing benchmarks. If it fails to predict the fresh gap, the method has failed.
Decisive Checks
The first check is the placebo. Erasing a non-target region must leave target-claim support near native levels, and erasing the target must leave anchor-claim support near native levels. Otherwise the curves reflect editing artifacts, and the method is invalid.
The second check is the proposal decomposition. For each search pipeline, $a_{\text{miss}}$ on V*Bench should exceed $a_{\text{miss}}$ on the fresh set if static scores are inflated. Their difference is our estimate of benchmark-specific non-perceptual credit. We expect it to be largest for pipelines whose search follows co-occurrence heuristics, since V* relies explicitly on such priors.
The third check, and the main one, is predictive validity. $D$ must rank models by their static-to-fresh gap better than the three existing diagnostics do. If it does, high-resolution leaderboards gain a per-claim correction that can be computed from the benchmarks they already have. If it does not, we will have shown that current high-resolution scores reflect perception more faithfully than the contamination literature would lead one to fear.
References
- CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
- V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
- Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG
- Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
- Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification
- Ask, Don’t Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement