Video anomaly detectors are increasingly trained on fake anomalies. These are events painted into normal surveillance footage by a diffusion model, a generator that learns to turn random noise into images by gradually removing it. The trouble is that a detector trained this way can learn to spot "pixels the generator touched" instead of "things that should not happen." Earlier work shows the same failure in simpler form. Detectors trained on cut-and-paste defects become boundary detectors (SPot-the-Difference), and diffusion-generated video anomalies carry inflated feature norms that detectors latch onto (PA-VAD). Generated video also flickers from frame to frame in ways real footage does not, which gives a detector a second, temporal shortcut.
We propose to remove these shortcuts directly. Alongside the fake anomalies, we generate fake normals: the same masks, the same diffusion pipeline, the same seams, but filled with ordinary scene content. A per-pixel discriminator then tries to tell real normal pixels from generated normal pixels using the detector's own features. The detector is trained so that it cannot win. We call the resulting features provenance-blind. Once the detector can no longer see where a pixel came from, the only signal left for separating anomalous from normal pixels is whether the content is anomalous. We build this on frozen single-pass diffusion features, because they are spatially precise enough to make per-pixel discrimination meaningful. We extend the discriminator to frame differences, so that generator flicker is removed along with spatial artifacts.
Core Claim
The synthetic-to-real gap in pseudo-anomaly training is mostly a provenance leak. It is local in space, as seams and blending halos are (DRÆM, DSR), and local in time, as generator flicker is. Our claim is that it can be closed by two things together. The first is crossed synthetic controls, meaning generated normal content produced under the same conditions as generated anomalies. The second is a dense discriminator that is conditioned on the class label and removes, from the detector's features, any information that separates real normal pixels from generated normal pixels.
The discriminator operates only within the normal class. That condition is what makes the idea work. Provenance is correlated with the anomaly label by construction, since every anomaly is generated. Removing provenance from all pixels would therefore erase the anomaly signal too. Requiring only that generated normal pixels be indistinguishable from real normal pixels does not.
Concretely, let $\phi(x)$ be the frozen multi-scale features of frame $x$, extracted from Stable Diffusion in one forward pass with no added noise, as in VPD. Let $A$ be a small trainable adapter and $H$ a per-pixel anomaly head. Let $y$ be the per-pixel anomaly mask and $p$ the per-pixel provenance (real or generated). Let $D$ be a U-Net discriminator that predicts $p$. The adapter and head solve
$$\min_{A,H}\; \mathcal{L}_{\text{seg}}\big(H(A(\phi(x))),\,y\big)\;-\;\lambda\,\mathcal{L}_{\text{prov}}\big(D(A(\phi(x)),\,\Delta_t),\,p\,\big|\,y=0\big),$$
while $D$ minimizes $\mathcal{L}_{\text{prov}}$ on the same normal pixels. Here $\Delta_t$ is the difference between adapted features at frames $t$ and $t-1$, and $\lambda$ weights the adversarial term. The difference channel lets the discriminator detect flicker without processing a full $H\times W\times T$ volume. That volume is the memory bottleneck the file flags for spatio-temporal dense discriminators.
Relation to Prior Work
Three papers motivate the proposal by documenting the leak. SPot-the-Difference shows that pasted-texture training produces boundary detectors, and softens seams with blurred alpha masks. DSR avoids image-space seams by perturbing codebook tokens in a quantized latent space. PA-VAD finds the magnitude bias in diffusion-generated video anomalies and corrects it with a global domain-adversarial module on pooled clip features, aligning real and synthetic features. We differ from all three.
We neither hide seams, as SPot-the-Difference does, nor avoid pixel space, as DSR does. Instead we make seams uninformative, by putting identical seams on normal content. Unlike PA-VAD, our alignment is per pixel, per frame, and conditioned on the class label, so it targets exactly the local artifacts these papers identify.
This also partly resolves the tension the file describes between feature-space synthesis (DSR) and pixel-space diffusion synthesis (Rai et al., PA-VAD). If provenance is removed, pixel-space generation keeps its realism without paying for its footprints.
Several papers supply the method. Rai et al. generate spatial pseudo-anomalies with an off-the-shelf latent diffusion model that inpaints masked regions of normal frames. We reuse that generator and add normal-content inpainting as the control. Their only discriminator is a global binary classifier on pooled ViFi-CLIP features, and it penalizes semantic shifts rather than provenance.
OASIS and the U-Net discriminator supply the full-resolution per-pixel discriminator design. OASIS's LabelMix, which blends real and fake regions along label layouts, is the natural regularizer for our discriminator's boundaries. VPD and Baranchuk et al. supply the frozen diffusion feature extractor.
Two papers constrain the design. FPDM found that jointly training diffusion prediction and refinement stages was unstable. That pushes us toward a frozen backbone and a small adapter rather than an end-to-end diffusion detector. TSAM aligns neighboring frames' features with optical flow, and the file notes that flow alignment breaks under exactly the chaotic motion anomalies produce. We therefore use raw feature differences rather than flow-warped ones, accepting some sensitivity to real camera motion.
Research Program
The work has three parts.
Crossed data engine. For each normal training clip we sample masks. Each mask is inpainted twice, once with an anomaly prompt and once with a scene-appropriate normal prompt, using the same model, the same seed schedule and the same blending. For temporal content we follow Rai et al.'s flow mixup to create temporal pseudo-anomalies. For their matched controls we apply mixup between two segments of normal motion. We also regenerate short normal clips with a video diffusion model, which provides flicker-bearing negatives.
Detector. The detector is a frozen Stable Diffusion U-Net queried at the clean, no-noise setting on each frame, followed by a lightweight adapter and a per-pixel anomaly head. Frame-level scores are obtained by max-pooling the pixel map.
Measurement. A provenance-probe protocol makes the central quantity measurable. After training, we freeze the adapter and fit a fresh linear probe to predict real versus generated on held-out normal pixels. We report the probe's AUROC as the amount of provenance left in the features. We also report PA-VAD's magnitude gap (the mean feature norm of generated minus real normal pixels) and the variance of the detector's score over time on regenerated normal clips.
The hypothesis predicts three things. Probe AUROC should fall toward chance under our training. Real-data AUC should rise as probe AUROC falls. And the correlation between the two should survive across backbones.
Baselines
The naive reference point is the same frozen-diffusion detector trained on Rai et al.-style pseudo-anomalies with no controls and no discriminator. It isolates how much the leak costs.
The closest published method is Rai et al.. It shares our generator and protocol and reports 86.61% AUC on CUHK Avenue, 71.65% on ShanghaiTech, and 85.37% AUC on UCSD Ped2 when trained only on synthetic UBnormal. The zero-shot Ped2 number is the fairest single target for the synthetic-to-real claim. We rerun their released configuration where possible. Otherwise we use our reimplementation with their reported numbers shown alongside.
The strongest one-class numbers are from DDL: 90.35% AUC on Avenue, 98.46% on Ped2 and 74.25% on ShanghaiTech. UBnormal proxy training reports 93.0% on Avenue. Beating DDL on Avenue and ShanghaiTech under the same one-class-plus-pseudo-anomaly protocol would count as a clear win. Ped2 is near ceiling and serves only as a check that nothing breaks.
PA-VAD reports 98.2% on ShanghaiTech and 82.5% on UCF-Crime, but under weak supervision with video-level labels on real data. We do not claim to beat it under a different protocol. Instead we transplant its DARM-style global domain discriminator onto our detector as an ablation.
All learned variants share the same generated data volume, adapter size, training steps and a fixed hyperparameter search budget: eight configurations, selected on a held-out split of synthetic validation data, never on test anomalies.
Ablations isolate each ingredient. The first uses crossed controls with no discriminator. The second uses the discriminator without crossed controls. The third replaces the dense discriminator with a pooled global one. The fourth drops the frame-difference channel. The fifth applies the discriminator to all pixels instead of only normal pixels, which we predict hurts. The sixth swaps the backbone for DINO or CLIP features at matched adapter size.
Why not just…?
Why not just add the synthetic normals as extra normal training data and skip the discriminator? This is the most serious threat. It may deliver most of the gain on its own, because it breaks the rule that every generated pixel is anomalous. It does not stop the detector from encoding provenance jointly with other cues, since generated pixels are still enriched for anomalies. The crossed-controls-only ablation, together with the provenance probe, settles the question. If that ablation matches the full method on real-data AUC and already drives probe AUROC to chance, we will report that the data design is the contribution and the adversary is unnecessary.
Why not just normalize feature norms, since PA-VAD's bias is about magnitude? Normalizing each pixel's feature vector to unit length is a cheap baseline and we include it. But magnitude is only one footprint. Seams, color shifts and flicker survive normalization. The probe measured after normalization will show whether provenance is still decodable. If it is not, the cheap fix wins and we say so.
Why not just use PA-VAD's global domain discriminator? A pooled discriminator can be satisfied by matching average statistics while local seams remain intact. SPot-the-Difference and DRÆM indicate the shortcut is local. The dense-versus-pooled ablation, scored on pixel-level localization, tests this directly.
Why not just use DINO or CLIP features, which are cheaper? The evidence for diffusion features in this setting is indirect. VPD shows better dense geometry than discriminative backbones on NYUv2 (RMSE 0.254) and ADE20K. SPot-the-Difference argues ImageNet features are trained to ignore small local changes. There is also a real risk that diffusion features are more sensitive to diffusion artifacts. The backbone swap is therefore a genuine experiment, not a formality, and a win for DINO would be reported.
Proposed Experiments
The primary experiment trains on the normal videos of Avenue, ShanghaiTech and Ped2 with crossed pseudo-anomalies and reports frame-level AUC. Where pixel annotations exist, we also report pixel-level localization AUC and AP.
The decisive synthetic-to-real test trains only on UBnormal, which is rendered in Cinema4D and includes pixel masks, and evaluates zero-shot on Ped2 and Avenue against Rai et al.'s 85.37% on Ped2.
A static sanity check applies the crossed-control recipe, without the temporal channel, to MVTec AD. The references are DRÆM's 68.4 pixel AP and DSR's 98.2% image AUROC. This tells us whether provenance blinding helps in the industrial setting where the shortcut was first documented.
The two checks that decide whether the idea is right are these. First, across all ablations and backbones, real-data AUC should increase monotonically as provenance-probe AUROC decreases. Second, score flicker on regenerated normal clips should drop only when the frame-difference channel is present. If probe AUROC falls but real AUC does not move, provenance was not the bottleneck, and the hypothesis is wrong.
References
- SPot-the-Difference Self-supervised Pre-training for Anomaly Detection and Segmentation
- Pseudo Anomalies Are All You Need: Diffusion-Based Generation for Weakly-Supervised Video Anomaly Detection
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- DSR - A Dual Subspace Re-Projection Network for Surface Anomaly Detection
- Unleashing Text-to-Image Diffusion Models for Visual Perception
- Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach
- You Only Need Adversarial Supervision for Semantic Image Synthesis
- A U-Net Based Discriminator for Generative Adversarial Networks
- Label-Efficient Semantic Segmentation with Diffusion Models
- Feature Prediction Diffusion Model for Video Anomaly Detection
- Progressive Temporal Feature Alignment Network for Video Inpainting
- Dynamic Distinction Learning: Adaptive Pseudo Anomalies for Video Anomaly Detection
- UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection