Segment-Masked Noise Prediction for Forgery Localization Without Forged Training Data

Most image forgery detectors learn what particular edits look like: the seams of a splice, the resampling traces of a resize, or the fingerprint of one generator. They lose much of their accuracy when a new editing tool appears. This proposal takes the other route and models only what authentic photographs look like. The model sees just the fine-grained sensor and processing noise of real images. Its job is to predict the noise that should sit inside one object-shaped region, given the rest of the photo and the region's own coarse content. Where the observed noise does not match the prediction, something was probably pasted, inpainted, or regenerated there. The model never sees a forgery during training, so it has nothing edit-specific to overfit to.

The second ingredient is where to hide pixels before predicting them. A general-purpose segmenter supplies whole object regions, and each region is hidden in its entirety. This matters because manipulations tend to follow object boundaries. If random patches are hidden inside a forged object, the visible forged neighbours reveal the forged noise statistics, and the predictor simply copies them. That is the "identical shortcut" that silences reconstruction-based detectors. Hiding whole segments removes this leak. In two sentences: train a model on real photos to predict a region's high-frequency noise from everything outside that region, and flag regions whose actual noise it cannot predict. Use segmentation masks to choose the regions, so a forgery can never serve as its own evidence.

Core Claim

Region-level masked prediction of a forensic noise field has three properties. It is trained only on authentic images. It is masked at the level of segmenter-proposed regions. It is scored by a calibrated likelihood rather than a raw pixel difference. Together, these should localize manipulations across splicing, GAN inpainting, and diffusion editing with a smaller cross-manipulation drop than supervised localizers. The claim is specifically about the gap between seen and unseen manipulation types, not about raw accuracy on in-distribution benchmarks, where supervised models will likely remain ahead.

Three design choices each answer a documented failure.

First, predicting a high-pass noise residual instead of RGB pixels answers the finding in On the Pitfalls of Using the Residual as Anomaly Score. That paper shows that pixel residuals mostly register brightness differences. They miss anomalies whose intensity matches normal tissue, and they fire on normal edges.

Second, whole-segment masking answers the same paper's result that high-capacity reconstructors perform worst because they reproduce the anomaly. It also answers the identical-shortcut concern raised in DRÆM.

Third, conditioning on the region's own low-pass content resolves a basic problem with reconstruction in natural scenes. The semantics of a hidden object cannot be predicted from its surroundings, but its camera-noise statistics largely can, once the model knows the object's coarse appearance.

How This Differs From the Closest Work

Each direction supplies one ingredient.

The mechanism comes from region-of-interest segmentation. RGMIM uses lung masks to decide which patches a masked autoencoder hides and reconstructs. We keep the idea that masks should decide what is hidden, with two changes. RGMIM applies it only as pretraining for a downstream classifier and relies on accurate organ masks. We apply it at inference time as the detector itself, with automatically proposed regions that may be wrong. The Role of Pleura and Adipose in Lung Ultrasound AI motivates the broader premise: restricting computation to the right region stopped a classifier from exploiting background shortcuts, raising AUC from 0.6553 to 0.7742. It also warns that hand-built region extractors break when the region is damaged. We test that fragility directly (see Research Program).

The method lineage is reconstruction-based anomaly detection. DRÆM learns a distance between an input and its reconstruction using synthetic Perlin-noise anomalies. We borrow its lesson that raw subtraction is a poor score, but replace the synthetic-anomaly discriminator with a predicted per-pixel variance learned from authentic data only. This avoids tying the score to a hand-designed anomaly generator. Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals is the nearest relative in spirit: a reconstructor trained only on genuine audio and probed at several random masking ratios. We replace random masking with segment-aligned masking and test whether that change is what matters.

The application regime is cross-manipulation localization. Three existing approaches come closest:

  • DisGRL and FakeDiffer reconstruct authentic faces and classify from the discrepancy. Both output image-level labels only, rely on face alignment, and FakeDiffer also trains a fake-face decoder.
  • GIFL reconstructs authentic DINOv2 features across general scenes and flags deviating regions. It is trained with forged data, has no region structure, and its authors report trouble with small forgeries and dense textures.
  • ForgDiffuser uses diffusion to generate the mask, not to model authentic content.

None of these masks whole regions, predicts noise instead of pixels or semantic features, or trains without forgeries.

Research Program

Noise field. We compute a fixed high-pass noise field $r$ for every image. The main choice is the Steganalysis Rich Model filter bank used by CFL-Net. We also compute a low-pass image $L$ that carries the scene's content.

Region proposals. A frozen, off-the-shelf promptable segmenter (the SAM2 family that Detective SAM builds on) produces masks for each image at several granularities.

Predictor. A vision transformer is trained on authentic photographs only. For each training image, we sample a segment $S$ and dilate it by a few pixels so that blending seams fall inside the hidden area. The model receives $L$ everywhere, the noise field $r$ outside $S$, and a mask token over $S$. It outputs, for each pixel $p$ in $S$, a predicted mean $\mu_p$ and variance $\sigma_p^2$. Training minimizes the Gaussian negative log-likelihood, and the same quantity serves as the per-pixel anomaly score:

$$ a_p = \frac{(r_p - \mu_p)^2}{\sigma_p^2} + \log \sigma_p^2 . $$

Here $r_p$ is the observed noise value at pixel $p$, and $\mu_p$, $\sigma_p^2$ are the model's prediction from context. The variance term addresses the edge-artifact problem. On authentic data the model learns that sharp edges and fine textures have inherently unpredictable noise, so it inflates $\sigma_p^2$ there. As a result, real structure no longer produces the false positives that plague raw residuals. At test time we run one masked pass per segment, batched across segments. Each pixel then receives the average score over the segments that contain it, and we compare this against taking the finest containing segment.

Why the rest of the image stays visible. The literature documents a real tension between restricting attention to a region and keeping global context. Forensic models such as SPAN and TransForensics show that long-range comparison across the whole image is essential. Our design keeps that comparison because the entire image outside the segment remains visible. It is the forged region's own interior that is withheld.

Stress tests of the design. Three analyses probe it:

  • Segmentation quality. We compare segmenter masks against ground-truth manipulation masks, which give an oracle upper bound, and against deliberately degraded masks: eroded, dilated, shifted, and merged with neighbours. This measures how much of any gain depends on segmentation quality, the upstream-dependency risk that the ultrasound work flagged.
  • Copy-move. We expect copy-move within a single photo to be the main failure mode, since the duplicated region carries the same camera's noise. We will measure this on the copy-move part of CASIA rather than hide it, and report whether pairing our detector with a global-attention model covers the gap.
  • Isointense edits. Following the pitfalls paper, we build edits whose average intensity matches the original region while the texture is replaced. This tests whether noise-field likelihood avoids the intensity blind spot that pixel residuals suffer from.

Baselines

Naive reference points. The first is a non-learned score: the local variance of the SRM noise field compared with the image-wide median, aggregated over the same segments. It shows what segment pooling of raw noise statistics achieves without any prediction. The second is the trivial prediction that marks every pixel as manipulated. It anchors F1 and IoU on benchmarks where forged regions are large.

Strongest existing method on unseen generators. On Forgery ADE this is GIFL. It reports an unseen-average F1 of 0.6703, with TruFor at 0.4983 and ManTra-Net at 0.4977 in the same table. We use the Forgery ADE split into seen and unseen generators exactly as published. GIFL trains on the seen generators; our predictor trains only on authentic images drawn from the same underlying scenes, so it sees no forgeries at all.

Beating GIFL's 0.6703 with zero forged training data is unlikely, and we do not claim we will. The success criterion has two parts. The authentic-only model should exceed the TruFor and ManTra-Net level of about 0.50 on unseen generators. And its seen-to-unseen drop should be clearly smaller than GIFL's, which we compute from a retrained GIFL if the file's reported numbers do not include seen-generator scores. A secondary row adds a small calibration head trained on seen forgeries on top of our scores. That row must exceed 0.6703 to claim an absolute gain.

Strongest method on modern text-guided editors. This is Detective SAM, which reports an average IoU of 36.99 and average F1 of 44.19 across its out-of-distribution editing suite, including MagicBrush, SIDA, CoCoGLIDE, UltraEdit, AutoSplice, and FLUX/Qwen benchmarks. We evaluate on the subsets that are publicly obtainable and, if needed, rerun Detective SAM on exactly those subsets. Beating it means higher average IoU on the shared subsets. Because Detective SAM itself notes weakness on splicing and copy-paste, we also report per-editor results. We do not want an average to hide where each method wins.

Real-world tampering. On IMD2020, NIST-16, and CASIA, CFL-Net reports AUC of 89.9%, 99.7%, and 86.3%. These are in-distribution results for a supervised model. We treat them as a ceiling for reference rather than as a target the authentic-only model must beat.

Diffusion mask generators. For ForgDiffuser, the file reports only relative gains over EC-Net. No absolute F1 on IMD2020 was found on Lacuna, so any comparison requires rerunning it. We will also report latency, since multi-step sampling is ForgDiffuser's known cost.

Ablations of the key ingredient. The central ablation holds the architecture, training data, masked-pixel fraction, and compute fixed, and changes only how pixels are hidden. The variants are:

  • random patches at matched ratio;
  • random patches at the multiple ratios used by the audio DiT probe;
  • rectangular boxes matched in size;
  • undilated segments;
  • dilated segments.

A second ablation replaces the noise-field target with RGB pixels. A third replaces the likelihood score with raw absolute residual. All models receive the same number of GPU hours and the same hyperparameter search budget, fixed in advance.

Why not just…?

Why not just train a supervised localizer on many synthetic edits, as Detective SAM does with its automated editing pipeline? This is the most serious competitor. Generating fresh edits with current editors may simply win on any benchmark whose editors resemble the training ones. The evidence against relying on it alone comes from its own authors: they report weakness on legacy splicing and copy-paste and sensitivity to perturbation strength. GIFL's table also shows the whole field shifting as generators change. Our experiments settle the question with a leave-one-editor-family-out protocol on the Detective SAM suite and Forgery ADE. If the supervised model's drop on the held-out family is as small as ours, the authentic-only premise fails, and we will report the method as a complementary signal rather than a replacement.

Why not just hide random patches, as masked autoencoders and the audio multi-ratio probe do? It is simpler and needs no segmenter. Our argument is that visible forged neighbours reveal the forged noise statistics. That argument is testable, and the masking ablation above tests it directly. If random masking at matched ratios performs within noise of segment masking, the segmentation direction contributes nothing, and we would drop it in favour of the simpler multi-ratio design. We also expect the gap to widen with forged-region size, since larger regions leak more to themselves under random masking. We will plot it that way.

Why not just pool an existing detector's heatmap over segmenter masks? Segment-level averaging alone can sharpen boundaries and remove speckle, so part of any gain could be pooling rather than prediction. We therefore apply the same segment pooling to TruFor-, CFL-Net-, and GIFL-style heatmaps and to the naive noise-variance map. We credit the prediction step only with improvements beyond those pooled baselines.

Why not just reconstruct RGB pixels, or use DRÆM's learned discriminator? The pitfalls paper shows that pixel residuals track brightness and fail on isointense, texture-level changes, and generative inpainting is exactly such a change. DRÆM's learned distance fixes the scoring but ties it to Perlin-noise anomalies that do not resemble photorealistic edits. The RGB-target and raw-residual ablations, together with a DRÆM-style discriminator trained with the same budget, measure both effects on the isointense edit set and on Forgery ADE.

Decisive Checks

The idea stands or falls on three results.

First, on unseen Forgery ADE generators and held-out editor families, the authentic-only segment-masked model must show a smaller seen-to-unseen drop than GIFL and the supervised baselines, while reaching at least the roughly 0.50 unseen F1 of TruFor.

Second, segment masking must beat matched random masking, with the gap growing as forged regions get larger. If it does not, the mechanism claimed here is wrong.

Third, the likelihood score over noise fields must hold up on isointense edits where RGB residuals collapse, and it must not raise false positives on high-texture authentic regions. We measure the latter as the false-positive rate on authentic images, stratified by local edge density.

Copy-move performance is expected to be weak, and we will report it as a known limitation rather than a result to be explained away.