Leave-One-Patch-Out Reconstruction for Localizing Image Edits

Modern editing tools such as diffusion inpainting and instruction-guided editors can change one object in a photo and leave every other pixel untouched. Detectors that look at the whole image then fail, because the authentic majority of the image drowns out the small edited region. Reconstruction-based anomaly detection offers a different route. A model trained only on real images redraws the input, and the places where the redrawing disagrees with the input are flagged. That route has a well-known failure of its own. A model that sees the whole input can simply copy the edited region back, so the disagreement vanishes (the "identical shortcut").

This note argues that the identical shortcut is exactly the gap between training error and cross-validation error. A reconstruction that sees a patch reports how well it can fit that patch. A reconstruction that must predict the patch from everything except the patch reports how well the patch agrees with its surroundings, and that agreement is what a forgery breaks. The proposed detector scores each patch by its leave-one-patch-out prediction error inside a single image. It borrows the machinery of approximate cross-validation (ways of estimating held-out error without refitting once per held-out point) to make that score cheap. The exact score needs one generator pass per patch. The approximations need a handful of passes.

Core Claim

Train a masked predictor on authentic images only. A masked autoencoder, meaning a vision transformer trained to fill in hidden image patches from visible ones, is the natural choice, since the file reports that MAE-based backbones are the strongest family on the NeXT-IMDL benchmark (NeXT-IMDL). For an image $x$ split into $N$ patches, define the leave-one-patch-out residual of patch $i$ as

$$r_i(x) = d\big(t_i(x),\; g_i(x_{-i})\big),$$

The symbols are as follows. $t_i(x)$ is a target description of patch $i$, either normalized pixels or a high-pass version that emphasizes sensor and compression texture rather than semantic content. $g_i(x_{-i})$ is the predictor's estimate of that target when patch $i$ and only patch $i$ is hidden. $d$ is a distance.

Computing $r_i$ exactly for every patch is leave-one-out cross-validation on the image's own patches, so it costs $N$ forward passes. The claim has two parts. The residual map $r(x)$ localizes edits better than full-input reconstruction residuals and better than random-ratio masking, especially on unseen editing tools and small edit areas. Structured approximations of $r(x)$ recover most of that accuracy at a cost of roughly $K \ll N$ passes.

Why the Cross-Validation Lens Is Load-Bearing

The approximations come from the cross-validation literature, and they transfer in a specific way.

Grouped holdout. The first approximation is the $K$-fold analogue. Partition the patches into $K$ groups whose members are spatially separated, for example a stride-$s$ lattice with $K=s^2$. Mask one group per pass. Every patch is then predicted from nearly its full surroundings, missing only far-away companions. This parallels the structured node-dropping of Approximate Cross-Validation for Structured Models, which defines held-out units on a graph with local dependencies and shows that approximate held-out losses track exact ones with under 1% error on spatial random fields. That paper supplies the method's logic. Here the "graph" is the patch lattice, and the dependency is the transformer's mostly local use of context.

Infinitesimal-jackknife correction. The second approximation is a first-order correction in the style of A Swiss Army Infinitesimal Jackknife. That work estimates the effect of dropping a data point by a Taylor expansion in per-observation weights instead of refitting. Here we attach a weight $w_j$ to each visible token's attention contribution. The estimate of hiding patch $i$ is $g_i(x; \mathbf{1}) - \partial g_i / \partial w_i$. Every self-derivative $\partial g_i/\partial w_i$ is obtained with $K$ Jacobian-vector products along group-indicator directions, which is again spatial grouping. Cross-terms between separated patches are treated as negligible.

The Swiss Army paper also constrains this design. It requires twice-differentiable objectives and warns about kinks and full-network Hessians. Our correction never inverts a Hessian, which sidesteps the scaling problem. Removing a token is not an infinitesimal change, however, so the linear estimate may be poor, and we test that directly rather than assume it.

Cavity analogy. Bayesian LOO approximations for Gaussian latent variable models computes a leave-one-out "cavity" posterior that removes a point's own influence. Our masked prediction is the deep, sampling-free counterpart: a reconstruction with the patch's own influence removed. That paper motivates the framing. It does not supply code.

We deliberately do not apply the jackknife to a linear detection head on frozen features, which the research file suggests. Exactly refitting such a head across folds takes seconds, so an approximation buys nothing there.

Relation to Prior Reconstruction Forensics

Three prior lines use reconstruction to expose forgeries.

Discrepancy-Guided Reconstruction Learning trains a discriminative pipeline on forged data. Learning Local Reconstruction Errors for Face Forgery Detection reconstructs faces from full inputs, so it remains exposed to the identical shortcut when blending is smooth.

Multi-ratio DiT residuals is the closest method. It freezes a generator trained on real audio and masks inputs at ratios 0.5, 0.75 and 0.9. Random high-ratio masking does defeat the shortcut, but it also deletes much of the authentic context each prediction relies on. That inflates residuals on real regions and produces the drowning problem in reverse. Our holdout pattern differs in a precise way: each patch is hidden alone, or with only distant companions, and every patch is scored.

DRÆM shows that passing the pair of input and reconstruction to a learned localizer improves pixel accuracy. We keep an optional variant that feeds $[x; r(x)]$ into a small head trained on synthetic edits, and we report the training-free map separately.

Research Program

Predictor. Start from a public MAE vision transformer. Fine-tune it with random masking on authentic images drawn from source pools that are disjoint from the benchmark test images (COCO, Flickr30k and OpenImages are the NeXT-IMDL sources). Two targets are trained: normalized pixels and a high-pass residual.

Score variants. We implement exact leave-one-patch-out, grouped holdout for $K \in \{4, 9, 16, 25\}$, and grouped holdout plus the jackknife correction.

Aggregation. Patch residuals are aggregated to an image score by a non-negative pooling in the spirit of Stay-Positive, such as a top-$k$ mean, so authentic patches cannot cancel suspicious ones.

Approximation diagnostics. For each $K$, we measure how closely the approximate map matches the exact one: per-patch Spearman correlation, relative error, and agreement of top-decile patches. We also record passes and wall-clock time per image.

Where the idea should break. The analysis asks two questions. First, is there an area-ratio threshold above which the edit exceeds the holdout context, so that neighbors are themselves forged and the leave-one-out residual collapses? Second, does the high-pass target outperform pixels on tools like FLUX, where low-level artifacts are reported to fade?

Baselines

Naive reference. The same predictor, fed the full unmasked image. This is exactly the in-sample residual, so beating it measures the identical shortcut directly. Any gain from the method must first appear here.

Random-ratio masking, matched budget. The multi-ratio DiT recipe (DiT residual probing), transplanted to the same MAE predictor at ratios 0.5, 0.75 and 0.9. We use the same number of forward passes as our grouped holdout, averaging residuals over random masks. This is the ablation that isolates the key ingredient: the structure of the holdout (every patch hidden once, with maximal context) versus generic masking. The DiT paper reports no numbers in the file ("no reported result found on Lacuna for multi-ratio DiT residuals"), so this baseline is run ourselves. Beating it means a higher pixel-level Average Precision (AP) at equal passes on cross-tool splits.

Supervised localizers. These are the strongest learned competitors. DeCLIP (DeCLIP) decodes vision-language features into forgery masks. A MaskCLIP-style localizer is the best family on NeXT-IMDL. Both are trained on NeXT-IMDL's training tools and tested under its cross-tool and cross-granularity protocols, with an equal hyperparameter search budget. The file gives no numeric NeXT-IMDL results ("no reported result found on Lacuna for NeXT-IMDL localization scores"), so all numbers are produced under one protocol. Beating these baselines means higher pixel F1 or AP on held-out tools; matching them in-domain is acceptable.

Image-level detection. Stay-Positive is the reference. It reports about 97–99% AP on 50%-inpainted images, where standard baselines fall to 6% AP, and 99.33 AP on FLUX against 57.25 AP for Corvi et al. (Stay-Positive). We rerun it on Semi-Truths area-ratio bins. The target is to match it on large edits and exceed it on the smallest area-ratio bins without training on any forgeries.

Industrial sanity check. DRÆM on MVTec AD reports 68.4 pixel AP and 98.0 image AUROC (DRÆM). We do not expect to beat it there. The check shows only whether the leave-one-out score transfers outside forensics.

Why not just…?

Why not just mask at random ratios, as in the audio DiT probe? This is the most serious threat, and it may work nearly as well when edits are large. Our argument is that high-ratio masks strip authentic context and raise false positives on textured real regions, which is the reconstruction analogue of signal drowning. The matched-budget ablation settles the question. If random masking ties grouped holdout on small-area Semi-Truths bins, the holdout structure adds nothing and we would say so.

Why not just train a supervised localizer on forged data? NeXT-IMDL reports severe performance collapse when test tools, semantics or granularities differ from training (NeXT-IMDL). Semi-Truths reports a 10% recall gap between two generators for one detector (Semi-Truths). A supervised model will likely win on tools it trained on. Our claim is confined to held-out tools, and the cross-tool protocol tests exactly that claim.

Why not just run exact leave-one-patch-out and skip the approximations? At 196 patches with a ViT-B, exact leave-one-out may be affordable offline, and then the approximations matter less. They matter at high resolution (1024+ patches) and for screening at scale. The fidelity-versus-cost curve tells practitioners which $K$ suffices. If $K=4$ already matches exact leave-one-out, the jackknife correction is unnecessary and we drop it.

Why not just note that inpainting is itself "predict from context," so the residual should be small? This objection is why the high-pass target exists. A context-conditioned generator produces plausible content, but a different model will not reproduce the original camera's noise and compression continuity. The pixel-versus-high-pass comparison on diffusion and FLUX edits tests this directly.

Decisive Checks

The idea stands or falls on three results.

First, at an equal number of forward passes, grouped leave-one-patch-out must beat random-ratio masking in pixel AP on NeXT-IMDL's cross-tool protocol and on Semi-Truths' smallest area-ratio bins.

Second, approximate maps at $K \le 16$ must correlate strongly with exact leave-one-out maps, with top-decile patch agreement high enough that localization AP drops only marginally. Failure of the jackknife correction alone is an acceptable, reportable outcome.

Third, the training-free score must retain its advantage on held-out tools where supervised localizers collapse.

If the first check fails, the contribution reduces to an efficient masking schedule and the forensic claim is withdrawn.