Vision-language models are scored on fixed image collections that leak into training data and get tuned against until leaderboard gains stop meaning much. Live benchmarks in text and code fix this by publishing fresh problems on a schedule and grading them with scripts. Vision has no equivalent that also tells you why a model failed. This proposal builds one. We start from real photographs that come with a scene graph, which is a structured list of the objects in an image and the relations between them, such as "cup on table" or "person holding umbrella". We make one small, deliberate change to that graph, for example swapping on for beside or deleting an object. An image-editing model then makes the same change to the photo. Questions are generated from the edited graph, so their answers are known exactly and can be graded by a script.
Every release draws new edits from the same pool of seed images. Each release is therefore fresh, while staying comparable to earlier ones. Each question is tied to one specific relation that changed, so a failure points at a specific perceptual weakness rather than a lower aggregate score. The open question is whether such a benchmark actually behaves better than the cheaper options. We ask whether its model rankings hold steady across releases, whether it resists models that were fine-tuned on earlier releases, and whether it exposes relation-level failures that static sets hide. The editing step adds image artifacts, which could make all of this go wrong, so the design is built around measuring that risk.
Core Claim
Two lines of evidence motivate the work. Procedural variation exposes brittleness that average scores hide. On DynaMath, which renders 10 programmatic variants of each of 501 math-plot seed problems, Claude 3.5 Sonnet scores 64.8% on average but only 35.3% when it must answer all 10 variants correctly. Separately, scene graphs give an exact, checkable account of image content: Explainable and Explicit Visual Reasoning over Scene Graphs reaches 100% on CLEVR when given ground-truth graphs. Our claim is that we can apply a minimal, verified graph edit to a natural image, render it, and confirm it, and that doing so yields three things. The first is an evaluation that can be refreshed and graded by script for real photographs, not only synthetic plots. The second is a family-level worst-case score, where a "family" is a seed image together with all of its edited variants. The third is a per-relation diagnosis that neither fresh-image collection nor text-only foils can supply.
The novelty lies in the edited image paired with an exact, script-checkable answer. It does not lie in procedural generation as such, since CLEVR already renders synthetic scenes from graphs. It also does not lie in graph-derived foils as such, since CREPE already writes those as text.
Relation to Prior Work
Three nearby lines of work each supply one ingredient and lack the others.
CREPE builds captions by walking Visual Genome scene graphs and writes hard negative captions by swapping atoms, swapping roles, or adding negations. It finds up to a 12% Recall@1 drop on unseen compounds and near-chance performance at 9 or more atoms. It motivates our work, but its images never change. A model can therefore detect a foil from how plausible its wording is, without looking at the image.
DynaMath supplies the variants-per-seed design and the worst-case metric. We reuse both, moving from plotted functions to natural scenes and relations.
A Benchmark for Compositional Visual Reasoning composes 9 elementary relations into 103 synthetic odd-one-out tasks. It reports that models trained on a composite rule fail on its individual parts. We test whether the same failure appears in modern models on real photographs.
For refresh mechanics we borrow from LiveBench, which replaces about a sixth of its tasks each month and grades only by script after documenting that model judges perform poorly on problems they cannot themselves solve. We also borrow from EvoCodeBench, whose freshly mined repositories keep the data leak ratio near 2%, against more than 41% for HumanEval. MAC is the closest live multimodal benchmark. It refreshes from new journal covers and reports that model performance collapses once embedded text is removed. MAC explicitly calls for evaluation that separates perception from reasoning, which is the gap we target. MAC refreshes by collecting new images and cannot control what changed between them; we refresh by controlled editing.
Adaptive Visual Scene Understanding: Incremental Scene Graph Generation is the closest work that pairs scene graphs with synthesis. It turns stored relation triplets into replay images with Stable Diffusion and pseudo-labels them with a frozen earlier model. That is a training method, and its labels inherit the old model's errors. Our answers come from the edited graph, never from a model's label. Its problem of labels inheriting errors still constrains us at the verification step described below.
Research Program
The generator starts from Panoptic Scene Graph Generation images. That dataset has 48,749 images with pixel-level object masks covering 133 classes and 56 non-overlapping relation types. We use it because masks, unlike bounding boxes, tell the editor exactly which region to change.
For each seed image we sample edits from a fixed menu. The edits are a predicate swap within a relation family (spatial, contact, or action), a role swap between subject and object, an attribute change, an object insertion, and an object deletion. Each edit is chosen to be physically plausible given the rest of the graph, for instance no insertion of a "cup on sky". An instruction-based editing model renders the edit. The MultiEdit dataset, with 107K samples across 6 editing tasks, is a candidate source for fine-tuning or choosing that editor. Questions come from templates that walk the edited graph using the four operations defined by the scene-graph reasoning work: find objects with an attribute, filter relations by type, follow a relation to its target, and combine sets with and/or/not. Answers are computed by running those operations on the edited graph.
Verification is the crux, because a rendered edit may not actually show the intended graph. An edit is kept only if three checks agree. The first is a geometric check using depth and mask positions, in the spirit of HSGG's geometry-guided reasoning, applied to spatial predicates. The second is an independent detector that confirms insertions and deletions. The third is a pixel-difference check that the change stayed inside the target region. A stratified sample of every release is also verified by human annotators. The automatic verifiers are themselves models, which creates a circularity: we might keep only edits that verifier-like models can see. We measure this directly in the experiments below.
Each release contains a fresh draw of edits and question templates over a rotating subset of seeds, plus a held-back set of never-released seeds.
Baselines
Baselines here are mostly alternative benchmark designs. They are judged by the same properties we claim for ours. All designs are scored on the same pool of candidate models, which spans frontier vision-language models, open 7–8B models, and a two-stage symbolic pipeline. The symbolic pipeline uses HSGG to extract the graph and the explicit reasoning modules of Explainable and Explicit Visual Reasoning over Scene Graphs to answer, a system that reports 67.5% on VQAv2.0. Every model gets identical prompts and decoding settings across designs.
The do-nothing reference is a static, graph-templated question set over the unedited PSG images. It has the same templates and answer engine as our benchmark but no edits and no refresh. The comparison against it isolates whether editing and refreshing add anything beyond templated questions. We beat it if our release-to-release rank correlation is at least as high while the gain from contamination, measured in the fine-tuning test below, is substantially smaller.
The strongest existing compositional design is CREPE-style text foils regenerated each release from the same graphs. Its reported effect is up to a 12% Recall@1 drop from seen to unseen compounds (CREPE). It is the strongest competitor because it already uses scene graphs and is cheap to refresh. We beat it if two things hold. First, a text-only model with no image access scores well above chance on CREPE-style foils and near chance on our image-edited items. Second, our per-relation error profile predicts model failures on held-out spatial benchmarks that CREPE's profile does not predict.
The strongest live-refresh design is MAC-style collection of fresh images with retrieval-option questions. MAC reports accuracies between 50% and 80% on MAC-2025, and its DAD description scaffold gives gains of 1% to 5% (MAC). We also evaluate DAD as a model-side baseline on our benchmark, to test whether a free-text description step closes the worst-case gap.
For the worst-case metric itself, the closest result is DynaMath's 64.8% average versus 35.3% worst-case for Claude 3.5 Sonnet. No reported result found on Lacuna for worst-case accuracy on natural-image relation variants.
The key ablation keeps everything else fixed and replaces graph-targeted edits with random regional edits that the image editor makes without reference to the graph. Answers in that condition come from re-extracting the graph from the edited image. This isolates whether the graph-directed minimal edit, rather than generic image perturbation, produces the diagnostic signal.
Why not just…?
Why not just resample CLEVR-style synthetic scenes each release? CLEVR can already be regenerated without limit from graphs, and it is the cleanest control we have. But with ground-truth graphs it is solved at 100% (Explainable and Explicit Visual Reasoning over Scene Graphs). It also tests rendered primitives, not the language-prior relation errors, such as "in" versus "beside", that HSGG documents on real images. We include CLEVR resampling as a design arm. If its per-relation model rankings match ours, the editing machinery is unnecessary and we will say so.
Why not just collect fresh natural images every month, as MAC and LiveBench do? Fresh collection solves contamination but gives no control over what varies between items. A score drop cannot be attributed to a relation, and no worst-case score over a family is possible. MAC's finding that models lean on embedded text (MAC) shows how uncontrolled content lets shortcuts in. This alternative remains a genuine threat for the narrow goal of contamination resistance alone. The fine-tuning experiment compares the two designs on exactly that axis.
Why not just use free-form questions graded by a strong model judge? LiveBench dropped model judges after finding that they misjudge problems they cannot solve themselves (LiveBench). Our answers come from the edited graph, so grading needs no judge.
Why not just write graph-based text foils and skip image editing? This is the CREPE-style design described above. Its weakness is that the image never changes, so it can be partly solved without looking. The blind text-only test in the experiments measures exactly that.
Proposed Experiments
The first decisive check targets artifacts, and the project depends on it. Each original image is passed through the editor with a no-op edit, and accuracy on these artifact-only images is compared with accuracy on the originals. If accuracy drops sharply, models are failing on artifacts rather than relations. In that case we either switch editors or restrict the benchmark to edit types whose no-op drop is negligible. Human annotators also rate a sample of kept edits as correct or incorrect, giving a measured error rate for the automatic verifier.
The second check targets verifier circularity. We correlate each model's score with how similar it is to the verifier models, comparing the automatically verified pool against the human-verified subset. A large gap means the filter is biased toward models that see the way the verifier does.
The third check targets stability. Across six simulated releases, we compute Spearman rank correlation of the model pool between consecutive releases for our design, the static baseline, CREPE-style foils, and fresh-image collection. The Benchmark Lottery reports six different top models across 70 four-task subsets of SuperGLUE, so we also measure top-1 churn across subsets of edit types.
The fourth check targets contamination resistance. We fine-tune an open 8B model on release k of each design. We then measure its gain on release k+1 and on never-released seeds, relative to the gain the same fine-tuning produces on the static set. A benchmark that resists overfitting should show a small transfer gain that reflects real skill.
The fifth check is diagnostic. We report worst-case accuracy per seed family and per edit type, test whether failures on composite edits (a role swap combined with an insertion) remain when the parts are tested alone, and check whether per-relation error profiles predict performance on GQA-Spatial and COCO-Spatial, the spatial benchmarks HSGG evaluates on. Each design is also run with a blind, text-only model to measure how much of it can be solved without the image.
The project succeeds if the no-op drop is small, the edited benchmark ranks models as stably as the static set, and it absorbs far less fine-tuning gain while staying near chance for blind models.
References
- DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
- Explainable and Explicit Visual Reasoning over Scene Graphs
- @ CREPE: Can Vision-Language Foundation Models Reason Compositionally?
- A Benchmark for Compositional Visual Reasoning
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
- MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
- Panoptic Scene Graph Generation
- HSGG: Training-Free Hierarchical Scene Graph Generation with Geometry-Guided Relation Reasoning
- The Benchmark Lottery