Forensic Checks That Rewrite Themselves: Execution-Refined Programs Against Detector Obsolescence

Detectors for AI-generated images and video usually stop working when a new generator is released. They learn the specific glitches of the generators they were trained on, such as the blinking errors of early face swaps or the grid-shaped upsampling traces of early GANs (generative adversarial networks, image generators trained against a discriminator). Newer generators remove those glitches. We propose to build a detector as a growing library of small, readable inspection programs instead of a single neural classifier. Each program measures something about a video or image, for example whether optical flow is consistent across a face boundary or whether shadow directions agree across objects. A language-model coding agent writes, runs, and debugs these programs. The feedback it gets comes from executing each program on a large corpus of real media and on a handful of samples from the new generator. When a new generator slips past the library, the agent writes new checks until one holds reliably on real media and fails on the newcomer. Checks that pass this test are added to the library and never retrained, so older checks keep working.

The idea joins two lines of work that have not met. Work on detector obsolescence has found that modeling "what real media looks like" generalizes better than modeling fakeness. Work on code refinement has found that language models fix programs much better when they get hard feedback from execution than when they reflect on their own output without it. The missing piece is an execution oracle for forensics, meaning an automatic judge that says whether a program is right. We supply one: statistical behaviour on a held-out real corpus becomes the judge of whether an inspection program encodes a true invariant of real media.

Core Claim

Our claim is that forensic knowledge can be kept current by an execution-refined program library. When a new generator appears and only $k$ samples of it are available (with $k$ around 10 to 50), a coding agent that refines inspection programs against a real-media oracle should recover detection of that generator. It should match or beat a neural detector fine-tuned on the same $k$ samples. It should not lose accuracy on earlier generators, and it should keep a controlled false-alarm rate on real media.

The acceptance rule is the core mechanism. Take a candidate check $c$, a program that maps a media item to a score, with its threshold $\tau_c$ set at the $(1-\alpha)$ quantile of its scores on a real calibration split. The check is accepted only if

$$ \mathrm{FAR}_{\text{real-heldout}}(c) \le \alpha \quad\text{and}\quad \mathrm{TPR}_{\text{new}}(c) - \max_{c' \in L}\rho(c, c') \ge \beta . $$

Here FAR is the false-alarm rate on a second, held-out real split. TPR is the fraction of the $k$ new-generator samples that the check flags. $\rho(c,c')$ is the agreement between the flags of $c$ and those of an existing library check $c'$, which penalizes duplicates. $L$ is the current library, and $\alpha, \beta$ are fixed tolerances. When a check fails the rule, the agent receives structured feedback: crash traces, the real items that produced false alarms, score histograms for real versus new samples, and the overlap with existing checks. It then revises the program. This is the same kind of rich, numerical failure signal that Execution-Grounded Agents found essential. Their Oracle-Guided Trajectory Search runs candidate programs against deterministic scientific oracles and reaches 84% pass rate against 58% for blind retries with Llama-3.1-70b. Our oracle is not a fixed reference implementation. It is a statistical test built from real data, which is new territory for that search method.

Relation to Prior Work

Two forensic papers motivate the gap. Multi-Agent Forensic Reasoning splits video forensics into Texture, Lighting, Motion and Physics agents. These are vision-language models, meaning models that read images and write text, and a Judge agent weighs their reports. On FaceVid-Forensics-100K it reaches only 53.28% F1 on 20 generators held out from training. Its "Physics agent" describes physics in words and never measures anything. CorrDetail documents that vision-language models invent forgery artifacts that are not there. It fixes this with self-correction questions and a high-frequency feature branch, all inside the network. Both papers keep verification inside neural text generation. We move verification into executed code whose outputs can be checked.

Two papers shape our design. Zero-Shot Detection of AI-Generated Images (ZED) models only real images, using the coding-cost gap of a lossless compressor. Natural Trace Forensics anchors detection on camera traces shared by real images. From both we take the principle that checks should encode properties of reality and be calibrated on real data. Both also mark a weakness to avoid. Pixel-statistic detectors lose ground under compression and on commercial generators: NTF falls from 96.2% mAP on lab data to 78.4% accuracy on Midjourney and Kolors. This steers our agent toward geometric, temporal, and object-level invariants rather than pure high-frequency statistics. The motivation also comes from Be More Specific. That work shows human realism judgments track localized object flaws: realism of the five largest objects explains overall realism with $R^2=0.62$.

On the code side, CodeTransOcean is a warning. Self-debugging on compiler errors lifts ChatGPT only from 48.57% to 52.57% over three rounds and then levels off. We test directly whether statistical, domain-level feedback escapes that plateau. Tool-growing agents such as LearnAct and OS-Copilot supply the pattern of keeping debugged functions as reusable tools. The developer–QA loop of MAGIS supplies the split between an author agent and a test-running critic. None of these systems applies the pattern to detection, where the correct answer is a statistical property rather than a passing unit test.

Research Program

We will build a sandboxed Python environment exposing standard vision libraries (OpenCV, SciPy, PyTorch, off-the-shelf optical flow, depth, landmark and segmentation models) as primitives. An author agent proposes checks. An executor runs each check over the calibration split, the held-out real split, and the $k$ new samples. A critic formats the acceptance-rule failures into feedback. As in Oracle-Guided Trajectory Search, several candidate programs branch per round and the most informative failure is kept for repair.

Accepted checks enter the library. The final verdict comes from a small logistic combiner over library scores. The combiner is refit on real data plus all fake samples seen so far, so it is the only trained component.

Generators arrive one at a time to simulate evolution. At each arrival we record three things. Detection of the newcomer tells us whether the library adapts. Detection of all earlier generators tells us whether the library forgets. Real false-alarm rate tells us whether calibration holds.

A secondary study asks whether checks built on geometric or temporal measurements are harder to attack than end-to-end networks. RAID shows those networks collapse under small adversarial pixel noise. Adversarial noise here means perturbations found by projected gradient descent, a method that nudges pixels along the gradient of a detector's output. Under that noise, white-box F1 falls to 0.0 and black-box AUROC drops from 0.99 to 0.67. We treat this claim with suspicion from the start. A program that cannot be differentiated only hides gradients; it does not guarantee robustness. So we attack it with transfer attacks and with differentiable surrogates of each check, not just gradient attacks on the combiner.

Baselines

The naive reference point is a frozen library: the checks that exist before a given generator arrives, with no new synthesis. It measures how much obsolescence actually happens.

The strongest existing method on the video benchmark is the multi-agent system of Multi-Agent Forensic Reasoning. It reports 53.28% F1 on the FaceVid-Forensics-100K out-of-distribution split. The same paper reports 47.45% for the best single vision-language model and 46.22% for Gemini-3.5-Flash. We will use its published split and evaluate on the same 20 held-out generators. Beating it means exceeding 53.28% F1 there while keeping real false-alarm rate at or below $\alpha$.

On images, the reference points are as follows. The frequency-domain ResNet-50 on MS COCOAI reports 0.80144 on binary detection and 0.44913 on attribution (A Comprehensive Dataset for Human vs. AI Generated Image Detection). ZED reports AUC above 95% on DALL-E 3, Midjourney v5 and SDXL. ZED and NTF will be rerun on our splits, including recompressed versions.

The competitor we consider most dangerous is a strong pretrained detector fine-tuned on exactly the same $k$ new-generator samples and the same real corpus, given a matched wall-clock and tuning budget. For this few-shot regime we found no reported result on Lacuna, so we will establish it ourselves.

Two ablations isolate the key ingredient. The first is one-shot generation: the agent writes checks once, and they are accepted or rejected with no revision. The second is compiler-only feedback: the agent sees only crash traces, never real-data statistics. The gap between the second ablation and the full system measures the value of the real-media oracle specifically. All agent variants share the same language model, the same budget of oracle calls, and the same primitives.

Why not just…?

Why not just fine-tune a neural detector on the new generator's samples? This is the real threat, and the experiments are designed around it. With only $k$ samples, fine-tuning is prone to overfitting and to forgetting older generators, while a library that only grows cannot forget. That is a hypothesis, not a result. If the fine-tuned detector matches us on both new-generator F1 and retention, the remaining advantage is only interpretability, and we will report it that way. We will also test the combination of fine-tuned features plus library checks. If that combination dominates, the honest conclusion is that programs complement networks rather than replace them.

Why not just ask a frontier vision-language model whether the video is fake? The file's evidence argues against it. Single models reach 46–47% F1 on unseen generators (Multi-Agent Forensic Reasoning), and they hallucinate artifacts (CorrDetail). Our comparison against the multi-agent system, run on the same backbone, directly tests whether measuring beats describing.

Why not just use a zero-shot real-image model like ZED and skip code entirely? For clean images this may be enough. ZED already exceeds 95% AUC on modern generators. The case for programs rests on two conditions: compressed and resized media, and violations of physical or semantic consistency that pixel statistics cannot see. We will report both methods on original and recompressed data. If ZED stays ahead under recompression, the program library is justified only for video and object-level flaws.

Why not just hand-write a fixed library of physics checks? A fixed expert library is exactly our "frozen library" baseline, and we seed the agent with it. The question is whether automatic refinement finds checks the seed lacks, once each new generator removes the flaws the seed targets. The gap between the frozen and evolving library across the arrival sequence settles it.

Proposed Experiments

The main experiment runs the arrival protocol on the 33 synthesis methods of FaceVid-Forensics-100K and on the five MS COCOAI generators. We use several random arrival orders, because the file gives no release chronology, and sweep $k \in \{10, 25, 50\}$. We report newcomer F1, retention F1 on earlier generators, real false-alarm rate, and oracle calls per accepted check.

The decisive check is conjunctive. The full system must beat the fine-tuned few-shot detector on retention at equal newcomer F1. It must also beat the compiler-only ablation by a clear margin. If either condition fails, the central claim fails.

A second experiment traces accepted checks per refinement round, to see whether statistical feedback avoids the three-round plateau reported by CodeTransOcean.

A third evaluates object-level checks on the OcR data from Be More Specific, against OLIP's reported 53.9% accuracy on predicting human realism scores. Examples of such checks are handle counts and symmetry.

Finally, we attack the library with transfer attacks from the RAID perturbations (RAID dataset) and with surrogate-gradient attacks on each check. We will report robustness only if it survives the surrogate attacks.