Same Content, New Pixels: Likelihood Audits That Separate Text from Layout Memorization in Document Models

When a model that reads document images scores well on a benchmark built from scientific papers, invoices, or reports, we cannot tell whether it read the page or remembered it. The model might have memorized the paper's prose during text pretraining. It might have memorized the page itself, meaning its pixels, layout and table geometry. Or it might have extracted the answer honestly. We propose a cheap audit that tells these apart. For each benchmark question, the audit compares how likely the model finds the correct answer under three versions of the input: the original page, the same content typeset again with a different font, column layout and template, and no page at all. A model that reads honestly should be roughly indifferent to how the page is typeset. A model that memorized the page should lose confidence when the pixels change but the information does not. A model that memorized the prose should still predict the answer with the image removed.

The tool comes from perplexity-based quality filtering, which scores text by how surprising a reference language model finds it. We apply it to a setting where it has been considered unreliable. DocMIA found that static loss signals mislead on document question answering, because answer difficulty and repeated exposure to the same page distort them. It therefore turned to costly test-time fine-tuning instead. Our central bet is that re-typesetting supplies a per-document reference that cancels those confounds. The same content in different pixels has the same difficulty and the same information. What remains in the likelihood gap is memorization tied to the specific rendering.

Core Claim

Let $a$ be the gold answer to question $q$ about a document page $I$. Let $\tilde I$ be a re-rendering of that page from its source, with identical text, tables and figures but a different typeface, margins, column structure and element placement. We define two per-instance scores:

$$s_{\text{pix}} = \tfrac{1}{|a|}\big[\log p(a \mid q, I) - \log p(a \mid q, \tilde I)\big], \qquad s_{\text{text}} = \tfrac{1}{|a|}\big[\log p(a \mid q, \varnothing) - \log p_{\text{ref}}(a \mid q, \varnothing)\big].$$

Here $|a|$ is the answer length in tokens, $\varnothing$ means the image is withheld, and $p_{\text{ref}}$ is the same model's no-image likelihood on matched documents published after its training cutoff. That reference is the temporal "clean baseline" idea of Estimating Contamination via Perplexity, carried over to answers about documents. The rendering-invariance score $s_{\text{pix}}$ measures how much the model prefers the exact pixels it may have seen. The blind score $s_{\text{text}}$ measures how much the answer is predictable from memory alone.

The claim has two parts. First, $s_{\text{pix}}$ detects contamination that comes from exposure to rendered pages, and it does so without gradients. In controlled experiments it should approach the detection quality of DocMIA's optimization-trajectory attack. Second, the pair $(s_{\text{text}}, s_{\text{pix}})$ correctly attributes contamination to the text channel or the pixel channel. That would let us settle, for documents, a disagreement in the literature. Both Text and Images Leaked! finds that multimodal contamination largely originates in the text-only backbone. Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations finds that vision-language models anchor memorization to visual features, so text-only probes miss it.

Relation to Prior Work

Three lines of work motivate the gap. Estimating Contamination via Perplexity supplies the method we reuse. It compares a model's perplexity on verbalized benchmark items against memorized and post-cutoff baselines, and finds strong leakage on QuAC, BoolQ, SQuAD_v2 and XSum. It is text-only, so it cannot distinguish what a model remembers about a page from what it remembers about prose. The two multimodal detectors above work with behavior rather than likelihoods. MM-DETECT shuffles multiple-choice options and masks words in back-translated captions. The semantic-perturbation work uses diffusion with edge-map control to edit natural scenes so that the correct label flips. Neither transfers to open-ended extraction from dense pages, and neither yields a continuous per-token score. Our re-rendering plays the role their diffusion edits play, but for documents. It is exact rather than generative, because we change presentation while holding content fixed, and it needs no image synthesis.

DocMIA both constrains and benchmarks the proposal. Its negative finding about static loss is the main reason to expect our approach could fail. Its trajectory attack, which measures parameter distance and convergence speed during brief fine-tuning, is the strongest existing detector on document question answering. Benchmarks built to resist contamination, AirQA and SciAssess, both acknowledge that models may already hold parametric knowledge of the source papers. This is the field problem our audit targets. The "blind test" in VeriSciQA, which prunes questions answerable from the caption alone, is the discrete ancestor of $s_{\text{text}}$.

Research Program

The first component is a re-rendering pipeline. For arXiv-derived material we recompile the LaTeX source under alternative document classes, fonts, margins and one- versus two-column settings. PIN-Arxiv pairs Markdown with page images and supplies this alignment at scale. AirQA's papers come from AI conferences, so their sources are largely recoverable. For scanned documents such as DocVQA, there is no source to recompile. We reconstruct each page from OCR text and boxes, then typeset it again in a new typeface, keeping reading order and table structure. Figures are a known limitation. Re-layout moves a figure but keeps its pixels, so $s_{\text{pix}}$ partly misses memorization of figure content. We report figure-centric questions separately, and we add a mild arm in which the figure is resampled and redrawn where the plot data is available.

Before using the pipeline, we must confirm that re-rendering does not itself hurt honest reading. On documents published after the cutoff, a clean model's accuracy and answer likelihood on $\tilde I$ should match those on $I$ within a calibrated band. We estimate this band per model and per document type. Every later threshold on $s_{\text{pix}}$ is expressed relative to it.

The second component is a controlled contamination study, which gives us ground-truth labels. We continue pretraining an OCR-free document model in the style of Pix2Struct on a PIN-Arxiv subset. Into that subset we inject benchmark source papers at known exposure counts of 1, 3 and 10, through three channels. In the text-only channel the model sees the Markdown. In the pixel-only channel it sees the rendered pages under its screenshot-parsing objective. In the third channel it sees both. A parallel arm fine-tunes Donut and Pix2Struct on DocVQA with a member/non-member split that follows DocMIA's protocol, so the two methods can be compared directly. The attribution claim succeeds if $s_{\text{text}}$ rises mainly under text exposure and $s_{\text{pix}}$ rises mainly under pixel exposure, and if both scores grow with exposure count.

The third component is a field audit of released open document and vision-language models on AirQA, SciAssess and DocVQA. Each model is calibrated against papers posted after its cutoff, matched by venue and topic. The audit reports, for each benchmark and model, an estimated contaminated fraction and the dominant channel.

Baselines

The do-nothing reference is the raw conditioned answer log-likelihood $\log p(a \mid q, I)$, thresholded per model. We expect it to perform near chance, and DocMIA's account of distorted static loss predicts as much. We include it because it is exactly our score with the re-rendered reference removed. That makes it the ablation that isolates the key ingredient. If it matches $s_{\text{pix}}$, the re-rendering contributes nothing. The second ablation removes the no-image arm, so we can measure how much attribution depends on $s_{\text{text}}$.

The strongest existing method is DocMIA's trajectory attack. On DocVQA it reports up to 82.5% F1 against Donut and about 72% F1 against VT5 and Pix2Struct, and 11% true-positive rate at 3% false-positive rate against Pix2Struct-Base (DocMIA). We run it on our member splits with its published settings, in both the white-box and the proxy-distillation black-box variants. Our method receives the same member and non-member documents and the same validation data for setting thresholds. We report wall-clock and GPU-hours for both, since DocMIA needs gradient steps per document and our audit needs three forward passes. Beating it means equal or higher F1, or higher true-positive rate at low false-positive rate, at a small fraction of the compute. Matching it within a few F1 points at an order of magnitude less compute would also count as a practical win, provided we report it as such.

The text-perplexity temporal audit of Estimating Contamination via Perplexity is run on OCR text of the same pages, with the same post-cutoff reference pool. The blind-accuracy test in the style of VeriSciQA asks whether the model answers correctly without the image. On the few multiple-choice subsets, we also run MM-DETECT's option-order test (Both Text and Images Leaked!). We found no reported result on Lacuna for any of these baselines on DocVQA, AirQA or SciAssess, so all numbers for them will come from our own runs.

Why not just…?

Why not just ask the model the question without the image? This is the most serious threat. For contamination that enters through the text backbone, which MM-DETECT argues is common, a blind test may catch most of it, and our $s_{\text{text}}$ is a continuous version of the same idea. The blind test is structurally unable to see pixel-channel memorization, because a model that recognizes a page it has seen still needs the page in front of it. The controlled pixel-only arm settles the question. If blind accuracy and $s_{\text{text}}$ detect pixel-only exposure as well as $s_{\text{pix}}$ does, the decomposition is unnecessary, and we will say the simpler test suffices.

Why not just apply text perplexity with temporal baselines to the OCR'd page? It measures memorization of prose, not of pages. The semantic-perturbation study found that text-side probes fail to separate contaminated vision-language models whose memorization is anchored in visual features (Contamination Detection for VLMs). OCR serialization of tables and formulas also inflates perplexity in ways unrelated to memorization, a known failure of text-only scoring (CCNet). The same pixel-only arm tests this baseline directly.

Why not just use DocMIA? On document question answering it is the method to beat, and it may remain more accurate. It needs gradient access or a distilled proxy, plus optimization for every document. That cost rules it out for auditing many models across thousands of papers, and it outputs a membership bit without saying which channel leaked. The head-to-head comparison on its own protocol shows whether a forward-pass score gives up too much accuracy to be worth the savings.

Why not just decontaminate the corpus with n-gram matching? That works when you own the corpus, and for our controlled study we do. It does not work for released models with undisclosed data, and it misses reformatted or translated copies (Data Contamination Can Cross Language Barriers). The audit targets the case where the corpus is unknown.

Decisive Checks

Three results would decide the proposal. First, the clean-model invariance check must hold: on post-cutoff documents, the gap between likelihood on the original and the re-rendered page must stay near zero. If re-rendering alone moves clean models substantially, $s_{\text{pix}}$ is measuring robustness to typesetting rather than memory, and the idea fails. Second, in the controlled study, $s_{\text{pix}}$ must beat the raw-loss ablation by a wide margin, its detection AUROC must rise with exposure count, and it must respond more to pixel-only exposure than to text-only exposure, while $s_{\text{text}}$ shows the opposite pattern. Third, on DocMIA's DocVQA protocol, our F1 and our low-false-positive true-positive rate are compared against the reported 82.5% for Donut and roughly 72% for Pix2Struct, with compute logged for both methods. If these checks pass, the field audit on AirQA and SciAssess yields the first per-channel contamination estimates for scientific document benchmarks. It would also show directly whether the residual knowledge of source papers that those benchmarks' authors worry about comes from the text a model read or the pages it saw.