Counterfactual Re-Rendering for Contamination-Aware Document Benchmarks

When a vision-language model answers a question about a scanned receipt or a financial table, we usually assume it read the page. Often it did not need to. Public document benchmarks such as DocVQA, FUNSD and CORD have been online for years. A model may have memorized the page image, the answer key, or the underlying facts during text pretraining. This proposal turns each existing benchmark item into a small family of rendered variants. Each variant is regenerated from the item's structured content. One keeps the content but redraws the layout. Another keeps the layout but swaps the facts so that any memorized answer becomes wrong. A third prints the same words as a plain strip of text with no layout. Comparing a model's answers across these variants separates three things a single accuracy number mixes together: reading the page, recalling memorized content, and using the page's visual structure.

The obvious objection is cost. Document evaluation is already expensive because each question requires encoding high-resolution page images, and multiplying every item into five variants makes it worse. The second contribution therefore adapts benchmark-shrinking methods from item response theory (a statistical model from educational testing that estimates each question's difficulty and each test-taker's ability) to this setting. The adapted model chooses a small set of items and variants that still recovers each model's full profile. It uses only answer correctness, not output log-probabilities, so it also works on closed APIs.

Core Claim

A paired, procedurally re-rendered version of a public document benchmark yields per-model estimates of content memorization, image memorization, layout reliance and visual-context reliance. These estimates rest on a controlled comparison rather than a heuristic perturbation. A correctness-only response model over items and variant types can estimate them from roughly a tenth of the variant evaluations.

The core mechanism is the counterfactual re-render. Re-rendering alone is not enough, because a model that memorized "Company X's 2021 revenue" will still answer correctly when the table is redrawn in a new font. Only counterfactual content separates reading from recall. Layout re-rendering and flat rendering then identify what kind of reading the model does.

Positioning Against Prior Work

The motivating evidence comes from contamination studies. Both Text and Images Leaked! uses option-order sensitivity and slot guessing to show that multimodal benchmarks leak, and that much of the leakage comes from the language model's text pretraining. That finding is exactly why content counterfactuals matter more than image perturbations. Work on rephrased samples and reasoning models, namely Rethinking Benchmark and Contamination for Language Models with Rephrased Samples and On The Fragility of Benchmark Contamination Detection in Reasoning Models, constrains the design. Both find that perturbation-based detectors are brittle. This proposal therefore defines contamination by an outcome with a known correct answer, and avoids relying on a perplexity or sensitivity shift.

On the document side, DISCO: Document Intelligence Suite for COmparative Evaluation compares OCR pipelines with direct visual question answering and finds that direct answering wins on single-page documents. It cannot tell whether that win reflects better spatial reading or parametric memory, and the flat-strip and counterfactual conditions answer exactly that question. ViDoRe Benchmark V2 reduces extractive shortcuts with blind querying, but it remains a static, crawlable collection.

The rendering method is borrowed from two places. RIDGE generates structured text in a hierarchical format and places it into realistic forms with a learned layout model. It does this only to augment training data and never pairs synthetic documents with originals for evaluation. The flat-strip condition borrows the rendering of SEETOK (See the Text), which draws text as thin horizontal images for a vision encoder. That condition holds the input in pixels while removing layout, so "visual context loss" is measured without also switching from pixels to tokens.

The scalability component builds on tinyBenchmarks, DISCO: Diversifying Sample Condensation, and multimodal IRT (M3IRT), which splits item difficulty into text-only, image-only and cross-modal parts. These methods shrink multiple-choice language or general vision-language benchmarks. Several rely on log-probabilities. None of them models a structured axis of variants per item.

Research Program

The first component is the re-rendering engine. For forms and receipts (FUNSD, XFUND-ZH, CORD), the annotations already give word boxes and key–value links. For a filtered subset of DocVQA, we recover structure with OCR plus layout parsing and keep only items whose answer is a recoverable span or a quantity computable from spans. We will report the coverage this filter achieves.

From each item's structure we render five conditions:

  • O, the original image.
  • R0, a faithful re-render that reproduces the original geometry with the engine's fonts.
  • Rc, which keeps the same layout but substitutes values with type-matched counterfactuals. Arithmetic constraints are re-enforced, for example recomputing a receipt total after swapping line prices, and the gold answer is updated.
  • Rl, which keeps the original content but draws a new layout produced by a RIDGE-style generator conditioned on the same hierarchy.
  • Rf, which renders the same content as a flat reading-order strip.

A plain-text OCR condition T is added as the classic pipeline reference.

The differences between conditions define the diagnostics:

  • O − R0 isolates memorization of the specific image, plus any renderer artifact. The artifact part is estimated separately on held-out documents created after the models' cutoffs.
  • R0 − Rc isolates content recall.
  • R0 − Rl measures dependence on one particular layout.
  • R0 − Rf measures how much the model uses visual structure at all.

The second component is a correctness-only response model. Each model $j$ gets an ability $\theta_j$ and loadings $\lambda_{jk}$ on each condition contrast $k$. Each item $i$ gets a difficulty $b_i$ and a susceptibility $a_{ik}$ to contrast $k$. The probability that model $j$ answers item $i$ correctly under condition $k$ is $\sigma(\theta_j + a_{ik}\lambda_{jk} - b_i)$, where $\sigma$ is the logistic function. Item parameters are fit on a pool of historical open models. A new model is then evaluated on a selected subset of (item, condition) pairs, chosen to maximize the information gained about $\theta_j$ and the $\lambda_{jk}$. The full-suite estimate is read off the fitted model.

The third component is validation by planted contamination. We fine-tune an open vision-language model on a known random half of the test images and answers. We then check whether the per-item R0 − Rc gap and the O − R0 gap recover which items were planted.

Baselines

The naive reference point is the original static benchmark score, condition O alone. If our diagnostics never change the ranking of models or the interpretation of their scores relative to O, the method adds nothing. For CORD, the strongest reported reference is the RIDGE-augmented Qwen2-VL-7B at an ANLS of 85.53% (RIDGE). No reported result was found on Lacuna for an aggregate DocVQA score of a comparable model, so we will measure O ourselves for every model.

The strongest existing contamination detector is the MM-DETECT family of option-order sensitivity and slot-guessing tests (Both Text and Images Leaked!). It is the only method in the file designed for multimodal contamination. It will be run on the same models and items with its published protocol, adapted to extractive answers by masking answer slots in captions or questions. No reported number for this detector was found on Lacuna for document benchmarks. To beat it, our gap statistic must achieve higher area under the curve than this detector at identifying planted items, at equal query budget.

For the scalability side, the strongest competitors are DISCO sample condensation and tinyBenchmarks' gp-IRT estimator. DISCO reports a mean absolute error of 1.07 percentage points and a Spearman correlation of 0.987 on MMLU with 100 samples (DISCO). tinyBenchmarks reports an average error of 1.9% and Spearman above 0.95 on MMLU with 100 examples (tinyBenchmarks). M3IRT reports recovering rankings with 10% of questions at 13% of inference cost (M3IRT). These methods are given the same model pool, the same total evaluation budget counted in rendered page encodings, and the same flattened (item, condition) pool. DISCO is restricted to correctness signatures, because many target models are closed. We beat them if, at equal budget, our estimates of the per-model contrasts $\lambda_{jk}$ and full-suite accuracies have lower error on held-out model families.

Two ablations isolate the key ingredients. The first removes Rc, which tests whether layout re-rendering alone can detect content recall; we predict it cannot. The second replaces Rf with plain text T, which tests whether the modality switch, rather than layout removal, explains the visual-context gap.

Why not just…?

Why not just swap values in the OCR text and test a text-only model? This is the cheapest content counterfactual, and for pure content recall it may come close to Rc. The planted-contamination experiment will report both. It cannot test image models on the image they are actually evaluated with. It also cannot measure layout or visual-context reliance, which text-only representations discard (DuReadervis, Spread2RML). If text swaps match Rc on content recall, we will say so and keep rendering only for the layout contrasts.

Why not just build a fresh fully synthetic benchmark with RIDGE? This is the most serious alternative. A fresh benchmark avoids contamination but loses the per-item link to the public benchmark everyone reports on. It therefore cannot say how much of a model's DocVQA score is memorized. Synthetic documents may also be easier or differently distributed. We test this directly by comparing model rankings on unpaired synthetic documents against rankings on Rc. If unpaired synthetic documents give the same rankings and the same contamination conclusions, the paired design is unnecessary for ranking, though not for per-item diagnosis.

Why not just subsample randomly to control cost? tinyBenchmarks reports that random subsets give high-variance rankings. M3IRT shows that random samples are full of shortcut items. Random sampling of (item, condition) pairs is included as a budget-matched curve, so the answer is measured rather than assumed.

Why not just use perturbation detectors like option-order sensitivity? Document questions are mostly extractive, not multiple choice. Perturbation detectors are also known to be fragile (On The Fragility…). The planted-contamination test compares the two approaches head to head.

Proposed Experiments

We evaluate a pool of open vision-language models of varying size and pretraining cutoff, plus several closed APIs, on FUNSD, XFUND-ZH, CORD and filtered DocVQA under all six conditions. Metrics are ANLS for answers and F1 for entity linking.

The first decisive check is planted contamination. Detection is reported as area under the curve over planted items, for our gaps versus the MM-DETECT-style baseline. Results are also stratified by whether contamination was image-plus-answer or text-only answer, mirroring the finding that leakage often comes through text.

The second check is a temporal control. We collect documents produced after all models' training cutoffs and confirm that the O − R0 and R0 − Rc gaps are near zero on them. This bounds renderer artifacts.

The third check is efficiency. We hold out entire model families, fit item parameters on the rest, and plot estimation error of the full-suite accuracy and of each contrast against the evaluation budget. The curves compare our method, DISCO, gp-IRT, M3IRT and random sampling.

As a secondary extension, we apply Rl and Rc to ViDoRe pages with ColPali, which reports 81.3 nDCG@5 (ColPali), and with ColModernVBERT, which reports 68.6 (ModernVBERT). This tests whether retrievers' scores depend on memorized page appearance.

The proposal fails if any of three things happens: content counterfactual gaps do not recover planted items better than existing detectors, the gaps are not near zero on post-cutoff documents, or condensed estimates are no better than random subsets of the same size.