Tracking by Meaning, Judging by Evidence: Forgery Localization in Long Compressed Video

When a video is uploaded to a social platform, it is recompressed. That recompression erases most of the faint pixel-level traces that detectors of AI-edited content rely on. A video detector could in principle make up for weak per-frame evidence by looking at many frames. The usual way to do that has a hidden flaw, though. Memory-based video models carry the model's past predictions forward and let them shape the next frame's prediction. Under heavy compression those predictions are noisy, so the noise feeds back on itself: an early mistake is stored in memory, trusted, and amplified, and the mask drifts. The fix we propose separates two jobs. We follow where a suspicious region goes using coarse, meaning-level features that survive compression. We decide whether it is fake by adding up per-frame forensic evidence along that path, where each frame's vote is weighted by how badly the codec damaged that particular frame. Forensic predictions are never written back into the tracker's memory.

In two sentences: stop propagating fragile forensic predictions through time, and instead average them along trajectories found by compression-robust tracking. Weight each frame by a reliability score read from the codec, such as its frame type and quantization level, so that heavily damaged frames count less. If this works, errors average out over time instead of compounding, and long videos become an asset for detection rather than a liability.

Core Claim

Prior work in two fields points the same way. Forensics work shows that fine traces vanish under compression, while self-supervised backbones keep coarse information. Unveiling the Impact of Image Transformations finds that CLIP linear probes fall to 0.4% accuracy at JPEG quality 10, whereas DINO holds at 95%. GenImage finds that 64×64 downsampling drops ResNet-50 to about 57%.

Video segmentation work shows that feeding a model's own predictions into its memory is what causes long-horizon drift. Space-time memory models such as STM store predicted masks as keys and values. LVOS shows that replacing those stored predictions with ground truth recovers up to 20% J&F.

Put together, these results predict something neither field has tested. In compressed forged video, a memory model that propagates forensic masks will drift faster than it does on clean video, because each stored mask is both less accurate and more trusted than it should be. A design that keeps forensic outputs out of memory should not drift in this way.

Concretely, let $r$ index a tracked region and $t$ a frame. Let $s_t(r)$ be a per-frame forensic score for region $r$, calibrated as a log-likelihood ratio of fake versus real. Let $w_t \ge 0$ be a reliability weight predicted from codec side information and decoded-frame statistics. The region's accumulated evidence over a video of length $T$ is

$$L_T(r) = \sum_{t=1}^{T} w_t \, s_t(r),$$

and pixels are labeled forged when their region's $L_T$ crosses a threshold. If per-frame errors were independent, the noise in $L_T$ relative to its signal would shrink roughly as $1/\sqrt{N_{\text{eff}}}$, with $N_{\text{eff}}$ the effective number of weighted frames. This is the standard averaging argument.

Compression errors are not independent within a group of pictures, the run of frames predicted from one keyframe. The honest expectation is therefore that $N_{\text{eff}}$ grows with the number of such groups rather than the number of frames. Measuring that growth is part of the study.

Relation to Prior Work

Three papers supply methods we reuse.

Mask Selection and Propagation separates proposing masks from propagating them and uses a learned judge to decide what enters memory. Removing that judge cost 8 J&F points on DAVIS 2019 Unsupervised. We keep the separation but go further: forensic scores never enter memory at all.

Detective SAM adapts SAM2, a video segmentation foundation model, for single-image forgery localization using small adapters and deliberately discards SAM2's temporal memory. We turn that memory back on, but use it only to track regions, never to carry forensic labels.

TruVIL localizes video inpainting with 3D high-pass noise filters over short windows and is robust to H.264 at constant rate factors 18–28. We use it as a per-window evidence source and as the main baseline. It has no long-term memory, which is exactly what we add.

Two papers constrain the design. Lazy-XMem decides when to correct memory using the entropy of the model's own mask probabilities. We expect that signal to be misleading in compressed forged video, because blocking artifacts raise entropy whether or not the prediction is wrong. RA-Bench found that choosing which frames to inspect by motion gained only +0.99 AUC. That rules out designs that merely pick better frames and motivates designs that combine evidence across all frames.

Finally, ForgeryNet localizes forgeries in time, but only as clip-level start and end times with fixed-window models. It does not track forged pixels over long horizons.

Research Program

Tracker. A frozen SAM2-style tracker receives prompts from spatial proposals made by a per-frame forensic head. It links those proposals into region tracks using only its own appearance memory, which keeps the coarse, low-frequency cues that compression spares.

Forensic head. A lightweight adapter on a DINOv2 backbone produces $s_t(r)$ per frame. For inpainting, TruVIL's per-window noise stream is used instead. Both are calibrated on a held-out split with temperature scaling.

Reliability model. A small regressor predicts $w_t$ from two kinds of input. From the bitstream it uses frame type (keyframe or predicted), quantization parameter, and residual energy. From the decoded frame it uses blockiness and high-frequency energy. It is trained so that $w_t s_t$ is best calibrated on validation data.

We also build a variant that sees only decoded pixels, because re-transcoding on a platform overwrites earlier bitstream history. We expect this to be the realistic deployment setting.

Diagnostic study. Before any method comparison, we measure how forensic mask accuracy decays with frame index for a SAM2 memory model that propagates forensic masks. We compare the decay on pristine video and on video at several compression levels. This tests the premise that compression accelerates drift in forensic memory.

Testbeds. We use three.

  1. DVI2016 and DVI2017 under H.264 at constant rate factors 18–40, following TruVIL's protocol and extending it to harsher levels.
  2. A constructed long-horizon set, LVOS-Removal. We inpaint LVOS target objects using inpainting models trained on VOS2k5, as TruVIL does. The LVOS masks then serve as ground-truth forgery masks on minute-long videos with occlusion and reappearance, and we re-encode the results with H.264 and VP9 plus downsampling in the style of RA-Bench-LastMile.
  3. ForgeryNet face videos under its video perturbations, where the tracked region, a face, is naturally trackable.

For video-level detection, RA-Bench and RA-Bench-LastMile test whether accumulating evidence over whole clips helps against modern image-to-video generators.

Metrics. Pixel-level F1 and IoU, reported both overall and per temporal segment (the first, middle, and last thirds of each video), so that drift is visible directly. Robustness [email protected], the fraction of videos whose mean IoU stays above 0.5. Video-level AUC and fake recall on RA-Bench.

Baselines

The naive reference is a per-frame forensic head run independently on every frame, with no tracking and no accumulation. It shares the same DINOv2 adapter as our method. This isolates everything that time adds.

The strongest existing method for compressed video inpainting localization is TruVIL. It reports F1 of 0.82/0.89 and IoU of 0.61/0.72 on DVI2016 in its reported settings, with only a small drop across H.264 constant rate factors 18–28 (TruVIL). We run it with the authors' training on VOS2k5 and our identical compression pipeline. On LVOS-Removal we train it with the same data and schedule as our heads. Beating it means matching it within noise at constant rate factors 18–28 while exceeding it by a clear margin, which we set at 5 IoU points, at 34–40 and on the last third of LVOS-Removal videos. That is where a fixed-window method should fail and an accumulating one should not.

The strongest temporal competitor is a memory model that propagates forensic masks: SAM2 with Detective SAM-style adapters and memory re-enabled, fine-tuned end-to-end on the same compressed training videos with the same GPU-hour budget and hyperparameter search as our method. This is the design our claim says will drift. Beating it means higher late-segment IoU and a flatter IoU-versus-time curve.

A gated-memory variant adds Lazy-XMem-style entropy gating to that competitor, with no human clicks. This tests whether conventional uncertainty gating already solves the problem.

For video-level detection on RA-Bench, the file reports a mean AUC of 50.9% across seven traditional detectors on open-source generators and a fine-tuned multimodal LLM fake recall of 1.4% under LastMile (RA-Bench). Those are the reference points.

Three ablations isolate the key ingredients. The first uses uniform weights ($w_t = 1$), which tests whether codec-aware reliability matters beyond plain averaging along tracks. The second uses oracle tracks from ground-truth masks, which separates tracking error from evidence error. The third uses the pixel-only reliability model, which tests whether bitstream information is needed.

Why not just…?

Why not just run a robust per-frame detector and average its scores over the video? This is the real threat, because our method is a structured form of averaging. The evidence against it is partial. RA-Bench shows that frame-level discriminators collapse to near chance on modern generators, but that result concerns the detectors, not the averaging. For dense localization, fixed-pixel averaging blurs moving regions, while averaging along tracks does not. The experiments settle both points. The comparison with the naive reference plus pixel-wise temporal mean shows whether tracks matter. The uniform-weight ablation shows whether reliability weighting matters. If uniform track averaging matches the full method, we will report the simpler method as the finding.

Why not just train a memory model end-to-end on heavily augmented compressed video? Augmentation can make each frame's prediction better. It does not change the fact that the model reads its own stored predictions. LVOS's oracle experiments attribute up to 20% J&F to errors in memory contents even for models trained on the target domain. The end-to-end SAM2 baseline receives exactly this augmentation and the same compute. If its IoU-versus-time curve is as flat as ours, the premise is wrong.

Why not just gate memory writes with prediction entropy, as Lazy-XMem does? Entropy gating worked on clean natural video, but compression artifacts raise uncertainty everywhere, so a single entropy threshold may reject good frames and accept confident mistakes. Our gated-memory baseline tests this directly. We will also report how well entropy and our $w_t$ each predict per-frame forensic error, measured as rank correlation, on the same compressed frames.

Why not just use TruVIL's longer windows? Longer 3D windows increase compute and still do not reconnect a region after it reappears from an occlusion. The late-segment comparison on LVOS-Removal, which contains such reappearances, measures exactly this.

Decisive Checks

Three results would decide the proposal.

The first is the diagnostic. If forensic memory propagation does not decay faster under compression than on clean video, the motivation fails and we stop.

The second is late-segment IoU on LVOS-Removal at constant rate factor 34 and above. Our method should beat both TruVIL and the end-to-end memory model there, and the gap should widen with video length, which is the signature of accumulation replacing compounding.

The third is the uniform-weight and pixel-only ablations. They tell us whether the codec-aware weighting, the most specific new ingredient, earns its place, or whether decoupling tracking from judgment is the whole story.

A useful secondary check is whether $N_{\text{eff}}$ scales with the number of groups of pictures rather than the number of frames, which would confirm the correlated-error picture above.