Image and video generators trained on web-scraped data reproduce the flaws in that data. Stock-footage watermarks, burned-in captions, logos, and JPEG block grids all come back out. The training objective rewards matching every recurring pattern equally, so it cannot tell these flaws from content. The obvious fix borrowed from saliency research is to down-weight the training loss wherever a human would not look. It points the wrong way. Deep saliency models are built around "semantic attractors": faces, eyes, and especially text. A watermark is exactly the kind of thing that captures gaze. What separates an artifact from content is not whether it attracts attention. It is whether it is selected as content: a person describing the image looks at the "shutterstock" overlay and never mentions it.
This note proposes training a diffusion model with a per-pixel signal built from that gap. Pixels that are visually salient but not selected as content by a caption- or label-grounded recognizer, and that recur across unrelated images, are treated as unreliable at fine detail. They still contribute to training at high noise levels, where their exact appearance is drowned out. They are excluded at low noise levels, where the model would otherwise learn to draw them. In two sentences: artifacts are "attended but not described," and we can compute that mismatch without labels. We then use it as a spatially varying version of the noise-level "safe interval" that risk-sensitive diffusion derived for whole samples, so the generator never learns the fine score of the artifact.
Core Claim
Our main claim is about the localization signal. The mismatch between bottom-up attention and content selection, combined with how often a pattern recurs across a dataset, localizes structured training artifacts better than either saliency alone or recurrence alone. Our second claim is about the training rule. Using this map to set a per-pixel noise-level cutoff in the denoising loss reduces artifact inheritance more than masking the loss outright or cleaning outputs after the fact, and costs less image quality. The second claim depends on the first. If an oracle artifact mask fed through the same training rule does no better than simply inpainting the training images, the training rule is not the contribution and we will say so.
What the two literatures give us
The artifact-inheritance side motivates the problem and constrains the design.
Several papers document the phenomenon. VideoComposer reports that WebVid10M watermarks are baked into its outputs even though its motion-vector conditioning acts as a selector of moving content. MagicTime reports that time-lapse models confuse camera text overlays with physical change. Corvi et al. show that generators trained on LSUN reproduce 8×8 JPEG block boundaries and up-sampling grid peaks in the frequency spectrum.
The existing remedies each assume something these artifacts violate.
- SURE-Score learns a clean score from noisy measurements. It requires a known noise model, essentially Gaussian, and a logo is not Gaussian noise.
- MAD soft-thresholds an "extended score" at sampling time to pull trajectories back toward the clean data manifold. It is designed for small perturbations that point off the manifold, and it roughly doubles sampling cost. A high-contrast overlay is neither small nor unstructured.
- Risk-Sensitive Diffusion supplies the mechanism we reuse. With a known per-sample corruption level, there is a range of noise levels in which corrupted samples can guide training without biasing the clean distribution. Its "risk" is a property of a whole sample, and it is derived from known variances. The same paper also supplies a negative result we must respect: conditioning on a risk token and setting it to "clean" at inference over-represents the parts of the data that happened to be clean.
The content-selection side supplies the localization signal.
- Liu et al. show that deep features develop sensitivity to faces and text regardless of contrast. SALICON and DeepGaze show that recognition features explain most predictable fixations; DeepGaze II captures 88% of explainable information on MIT1003. Together these mean an attention model will flag watermarks, which is why plain saliency weighting fails.
- Don't Mention the Shoe! separates what is described from what is visible. Conceptual category is the main predictor of whether humans mention an object, with an F-score of 0.70 against a human ceiling of 0.74 on ImageCLEF 2015. This gives us the second half of the mismatch.
- Implicit Saliency shows that gradients from a classifier given a deliberately wrong label yield label-free saliency that is robust to blur. That robustness matters because our map must be computed on corrupted images.
Three bridging works sit closest. SADRE uses a saliency mask to heal invisible watermarks in single test images; it works after training on one image at a time. GazeFusion conditions Stable Diffusion on gaze maps as a user control and assumes clean weights. Sztukiewicz et al. show in classifiers that real artifact removal moves attributions off the shortcut while threshold tuning does not. That result motivates our use of attribution shift as a diagnostic, not only output metrics.
None of these computes an attended-but-unselected map, and none uses any saliency signal inside generative training.
Research Program
The mismatch map. For each training image we compute three ingredients, then multiply the first by the complement of the second and by the third:
- an attention map $s$ from a DeepGaze-II-style predictor;
- a content map $c$, the image regions a pretrained recognizer grounds the caption or class label to, plus the implicit-saliency gradient for the true label;
- a recurrence score $r$, which measures how consistently the local patch pattern at each position reappears across semantically unrelated images. We compute it from feature-space nearest-neighbor counts across captions or classes that share no content words.
The resulting per-pixel artifact probability is $a = s\,(1-c)\,r$. Each factor rules out a known failure. Attention alone flags faces. Content alone misses diffuse background. Recurrence alone flags legitimate common structure such as sky gradients, and it misses overlays that move or change text.
The training rule. Let $\sigma$ be the diffusion noise level. The per-pixel denoising-loss weight is
$$w_i(\sigma) = 1 - a_i\,\mathbb{1}[\sigma < \tau_i],$$
where $a_i$ is the artifact probability at pixel $i$, $\mathbb{1}[\cdot]$ is the indicator that the noise level lies below the cutoff, and $\tau_i$ is a per-pixel cutoff. Above the cutoff, a flagged pixel trains normally, because at that noise level its exact appearance is hidden. Below the cutoff, it gives no gradient, so the network must fill fine detail there from surrounding context and from other images where that location is clean.
We set $\tau_i$ from the artifact's estimated contrast, following the Risk-Sensitive Diffusion intuition that a corruption is harmless once the added noise exceeds it. This avoids the sampling bias of risk-token conditioning because nothing is conditioned at inference. It is also not plain loss masking, which leaves the score unconstrained at all noise levels in flagged regions.
Analyses. We test whether the FID tension in the literature resolves when FID is measured against the right reference. FloorPlan-LLaMa saw FID rise as artifacts were removed, while GazeFusion saw it fall. Our prediction is that FID against the corrupted training set worsens while FID against a held-out clean set improves. We also check that the generator's own attributions move off artifact regions, using the RRF and ADR metrics from Sztukiewicz et al. adapted to irregular masks.
Baselines
Naive training. The "do nothing" reference is standard denoising score matching (EDM recipe) on the corrupted data. It fixes the size of the inheritance problem. All methods share architecture, data, number of training images seen, and sampler. Each gets the same hyperparameter search budget, roughly ten configurations selected on a held-out clean validation split.
Data repair before training. This is the strongest practical competitor. An oracle-mask variant inpaints the known corrupted pixels before training. A realistic variant uses an off-the-shelf watermark/text detector plus diffusion inpainting. We also run SADRE-style reconstruction over the training images. No reported result found on Lacuna for any of these as training-data repair for generative models. The reference figures available are SADRE's per-image numbers on MS-COCO StegaStamp watermarks: bit recovery accuracy 0.40 against 0.72 for JPEG compression, at PSNR above 33 dB. Beating data repair means a lower artifact rate at equal or better clean-reference FID, specifically with the realistic, non-oracle masks.
MAD at inference. We apply MAD with its released code (delbraechter/MAD) to the naively trained model, tuning its attraction parameters a and b on the same budget. Its strongest reported number is on FFHQ trained with Gaussian noise: FID 134.42 for standard sampling, 74.52 with MAD. With pixel-removal corruption the figures are 387.87 and 104.00. These are for unstructured corruption. No reported result found on Lacuna for MAD on watermark or text-overlay corruption. Beating MAD means a lower artifact rate at matched FID with half its sampling cost.
Risk-Sensitive Diffusion with a whole-image risk score. We set each image's risk to the mean of its mismatch map. This competitor has the same intuition but no spatial resolution. Its reported gain is on MIMIC-III, where Wasserstein distance falls from 10.083 to 1.865, a different modality.
Ablations that isolate the key ingredient. First, an oracle-$a$ run that feeds ground-truth artifact masks through our training rule. Second, map variants that use attention alone, $1-c$ alone, recurrence alone, and every pairwise product. Third, uniform loss masking at all noise levels in place of the cutoff.
Why not just…?
Why not just weight the loss by saliency, as the obvious combination suggests? Because saliency models are tuned to fire on text and faces (Liu et al.). Down-weighting non-salient regions would protect the background and up-weight the watermark. The attention-only ablation tests this directly. We predict it inherits overlays at or above the naive rate. If it does not, the mismatch term is unnecessary and the simpler method wins.
Why not just detect watermarks and inpaint the training data? This is a real threat for known, fixed artifact types, and we treat it as the main baseline rather than a straw man. Its weaknesses are that detectors are type-specific: they do not cover JPEG grids or mosaics. Inpainting also introduces the inpainter's own artifacts into the data, a second-generation inheritance. If realistic detector-plus-inpaint matches us on watermarks, our claim narrows to artifact types without detectors. We will report that split explicitly.
Why not just use recurrence, for example a per-position median across the dataset? For fixed-position stock watermarks this may already suffice. This is precisely why recurrence is a factor in our map and an ablation of its own. The semantic factors should matter for overlays that move or change text, as in the MagicTime setting, and for legitimate recurring content that recurrence alone would wrongly suppress. If recurrence-only matches the full map on both, the content-selection direction contributes nothing here.
Why not just run MAD at sampling time? MAD's operator assumes small perturbations off the manifold, and it gives no way to tell a bright logo from a bright object. It also costs 1.5–2× sampling time. The head-to-head on structured overlays settles whether this matters in practice.
Proposed Experiments
Controlled corruption. We train on FFHQ and CIFAR-10 with synthetic corruptions where ground truth is known: fixed and randomly placed semi-transparent text watermarks, corner logos, and JPEG recompression at quality 20. CIFAR-10 uses class labels for the content map; FFHQ uses captions from a pretrained captioner. Artifact rate is measured three ways: an OCR/template detector on generated samples, residual correlation with the watermark template, and, for JPEG, the strength of the 8×8 spectral peak following Corvi et al. Fidelity is FID against the clean originals and against the corrupted set.
Real setting. We train on LSUN bedrooms for genuine JPEG inheritance. We train a small video model on a WebVid10M subset, using its captions for the content map, and count stock-watermark appearances with the same detectors.
Validating the map. Before any generative training, we measure how well the map localizes artifacts on the controlled sets (pixel AUC against ground-truth masks). We also verify that it does not suppress described content: on ImageCLEF 2015, regions the map flags should overlap rarely with objects humans mention.
Decisive checks. Three outcomes would falsify the proposal:
- the full map does not beat recurrence-only on moving-text corruption;
- the cutoff rule does not beat uniform masking given oracle masks;
- realistic detector-plus-inpaint matches us on every artifact type.
References
- VideoComposer: Compositional Video Synthesis with Motion Controllability
- MagicTime: Time-lapse video generation models as metamorphic simulators
- Intriguing properties of synthetic images: from generative adversarial networks to diffusion models
- Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data
- MAD: Manifold Attracted Diffusion
- Risk-Sensitive Diffusion: Robustly Optimizing Diffusion Models with Noisy Samples
- Predicting eye fixations using convolutional neural networks
- SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks
- Using Deep Features to Predict Where People Look
- Don't Mention the Shoe! A Learning to Rank Approach to Content Selection for Image Description Generation
- Implicit Saliency In Deep Neural Networks
- Saliency-Aware Diffusion Reconstruction for Effective Invisible Watermark Removal
- GazeFusion: Saliency-Guided Image Generation
- Investigating the Relationship Between Debiasing and Artifact Removal using Saliency Maps
- FloorPlan-LLaMa: Aligning Architects' Feedback and Domain Knowledge in Architectural Floor Plan Generation