Two Clocks for Multimodal Benchmarks: Dating the Data and the Reasoning Separately

"Live" benchmarks try to keep test questions out of a model's training data by using material published after the model's training cutoff. Multimodal live benchmarks, which test models that read images as well as text, apply this freshness only to the images. Their questions still follow the same question recipes as older, public benchmarks, and a frontier model often writes them. A chart drawn from last month's data, asked with a question template that has circulated online for years, is new in one sense and old in another. We propose a chart-reasoning benchmark that keeps two separate timestamps for every item: one for the underlying data and one for the reasoning program the question requires. The two are varied independently, in a two-by-two design. We also add an interrogation that a memorizing model cannot pass. We edit the data table behind each chart, redraw the chart, and check whether the model's answer moves to the new correct value.

This matters because the field's main signal for "this benchmark is clean" is breaking. In text, Test of Time: Rethinking Temporal Signal of Benchmark Contamination shows that questions an LLM generates from post-cutoff arXiv theorems show no performance drop after the cutoff. Post-cutoff physics questions even rose by 2.19 points, while cloze recall on the same papers fell by 3.83 points for GPT-4o-mini. Rephrasing launders memorized structure. No one has asked whether the same laundering happens in multimodal live benchmarks, where Both Text and Images Leaked! has already shown that models often answer visual questions from text they memorized, without using the image.

Core Claim

Our claim is that in chart and figure reasoning, contamination sits mostly in the reasoning structure and the text priors, not in the pixels. Refreshing the images therefore removes little of it. If true, three things follow. Accuracy on items with fresh data but old reasoning templates should track accuracy on fully old items. The real drop should appear only when the reasoning program itself is new. And a large share of "correct" answers on fresh-data items should fail to update when the data is edited.

The test we build is in the spirit of post-Turing evaluation. The critique in Turing Test Considered Harmful and the human-judge study Does GPT-4 pass the Turing test? argue that judging whether an output looks right rewards imitation over competence. A benchmark score is a weak version of the same problem. Our counterfactual test replaces "does the answer look right" with "does the answer depend on the evidence." That turns the Turing critique from a philosophical stance into a concrete metric.

Research Program

Every item is built from three things: a data table, a rendering specification, and a question written as a short executable program over the table. Because the table is the ground truth, answers are computed rather than judged, and any edit to the table yields a new verified answer. Questions are generated as programs, following Synthesize Step-by-Step and the visual-programming approach of SCOPE. Unlike SCOPE, we do not start from historical images that frontier models have likely already seen.

The data clock is set by sourcing tables in two ways. Pre-cutoff tables come from the Statista, Pew, and OECD sources underlying ChartQA. Post-cutoff tables come from timestamped public statistical releases, filtered per model by cutoff date as LiveCodeBench does for code.

The structure clock is set in two ways as well.

  • Old reasoning structures are programs mined from ChartQA's human-written questions. Each is converted to a canonical form: operator sequence, argument types, and output type.
  • New reasoning structures are directed acyclic graphs of operators, sampled with controlled depth and width as in DyVal. A sample is kept only if its canonical form does not appear among programs mined from ChartQA, Chart101, and SCOPE-Test.

The difficulty confound that Test of Time warns about is controlled in two ways. Old and new programs are matched on graph depth, width, and number of chart elements touched. The same table appears under both kinds of program.

Rendering is randomized across plotting libraries, marker styles, and gridlines. The perturbations in Plot Twist show these alone can move accuracy, so they must be averaged over rather than left as hidden variables. Titles are either neutral or deliberately made to contradict the answer, since that paper found accuracy falls from about 79% to the 30% range once answer-bearing titles are removed.

The central measurement is the data-tracking rate. For an item $i$, let $x_i$ be the original chart and $x_i'$ the chart redrawn after a table edit chosen so that the correct answer changes from $y_i$ to $y_i' \neq y_i$. Let $f$ be the model's answer. Then

$$ \mathrm{DTR} = \frac{1}{|S|}\sum_{i \in S} \mathbf{1}\left[f(x_i') = y_i'\right], $$

where $S$ is the set of items the model answered correctly on the original chart, $f(x_i)=y_i$. A model that reads the chart scores near its perception accuracy. A model that answers from template priors scores near zero, because it repeats $y_i$. We also report a blind gap: accuracy with the image versus with the image replaced by a blank of the same size.

To check that the structure clock detects what it claims to, we inject a controlled leak in the style of MM-DETECT. LLaVA-1.5-7B is fine-tuned on a set of new-structure programs instantiated on different tables and chart styles. The structure axis should then report a matching rise in accuracy without a rise in DTR, while the data axis should report nothing.

Baselines

The naive reference point has two parts. The first is chance or majority-answer accuracy for each question type. The second is the blind model, which receives only the question text. Together they show how much of each cell's accuracy needs no image at all.

The strongest existing approach to freshness is temporal refreshing in the style of MMBench-Live. It harvests post-cutoff images and keeps the original task recipe, reporting a model-averaged correlation of r = 0.88 with MMBench and weaker memorization signals under PaCoST. Our post-cutoff-data, old-structure cell is a faithful re-implementation of that idea for charts. Its items are built in exactly the same way as ours: same tables, same rendering pool, same models, same decoding settings. The proposal beats it if the structure axis separates models, or reveals leaks, that this cell does not.

The strongest existing detection method is MM-DETECT from Both Text and Images Leaked!. It measures the accuracy change Δ under option shuffling and paraphrasing, and reports Δ = −5.3 for Claude-3.5-Sonnet on the ScienceQA training set. We run it on all four cells with the same models. Our method beats it if Δ fails to flag the injected structure leak while the structure axis and DTR do.

A perception-pipeline baseline extracts the table from the chart and passes it to a text-only reasoning LLM, as the ChartQA table-extraction systems did. It isolates how much of any failure comes from perception rather than reasoning. On ChartQA, the best reported overall accuracy is 45.52%. For a table-extraction pipeline on post-cutoff charts with novel programs, no reported result was found on Lacuna.

The key ablation removes counterfactual redrawing and keeps only the two clocks. It tests whether DTR adds information beyond plain accuracy differences between cells.

Why not just…?

Why not just trust MMBench-Live-style freshness? Freshness of images is exactly what is in question. Test of Time shows that LLM-generated questions on fresh material carry no decay signal while memorized structure persists. MMBench-Live itself notes it only refreshes the existing task distribution. Our old-structure, new-data cell measures directly how much this leaves behind. If that cell already matches the new-structure cell in difficulty and DTR, freshness is enough and we will say so.

Why not just use procedurally generated puzzles like PuzzleBench or CrossWordBench, which are contamination-free by construction? This is a real threat. Our answer is that synthetic grids mix novelty with an unnatural visual domain. PuzzleBench and CrossWordBench both show collapse that could be due to grid parsing alone; CrossWordBench reports r = 0.9358 between grid parsing and solving. Our design keeps realistic charts and real data while making structure novel. If the data clock turns out to carry no effect, the honest conclusion is that temporal sourcing is unnecessary. In that case the method reduces to counterfactual redrawing of static, table-backed charts, which is cheaper. The experiment decides which of these outcomes holds.

Why not just shuffle options and run blind baselines? These cheaply expose position bias and text leakage, and we include both. However, they cannot detect memorized reasoning templates. A model that knows how ChartQA-style questions resolve still uses the image to look up a value, so it passes the blind test. Paraphrase-based perturbation is the very laundering mechanism Test of Time identifies. The injected-leak experiment checks directly whether Δ misses what the structure axis catches.

Why not just ask a frontier model to judge reasoning traces? LiveBench finds that LLM judges fail on problems they cannot solve, and the Turing studies show judges reward fluent style. Our answers are computed from tables, so no judge is needed.

Proposed Experiments

We evaluate open models with documented cutoffs (Qwen2.5-VL-7B, InternVL2, LLaVA-1.5-7B) and proprietary models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro). Each of the four cells has roughly 2,000 items, and every item is paired with its counterfactual redraw.

The first decisive check is the interaction pattern. Our claim predicts a large structure effect, a small data effect, and DTR on old-structure items well below DTR on new-structure items matched for difficulty. The claim fails if accuracy and DTR depend on the data clock as strongly as on the structure clock.

The second check is the injected leak. The structure axis must recover the leak's size and MM-DETECT's Δ must not; otherwise our measurement adds nothing.

The third check is external validity. Model rankings on the new-structure cell are correlated with rankings on MATH-Vision, where GPT-4o reaches 30.39% against 68.82% for humans and 42.2% of GPT-4V's errors are reasoning errors. The new-structure cell should track this harder, reasoning-dominated benchmark better than the old-structure cell does.

Finally, we test the open question about step-by-step reasoning prompts in multimodal settings, where MATH-Vision found chain-of-thought sometimes hurts. We compare chain-of-thought with direct answering on DTR rather than accuracy, asking whether verbal reasoning makes answers depend more on the chart or only more fluent.