Forensic Program Synthesis for Unseen Image Generators

Detectors of AI-generated images work well on the generators they were trained on and often fail on new ones. A ResNet-50 that is 99.9% accurate on Stable Diffusion v1.4 falls to 54.9% on Midjourney (GenImage). Detectors built on a shared decoder fingerprint break when a generator changes its autoencoder, as with FLUX.1's 16-channel latent space (Aligned Datasets). The strongest adaptive method, PROBE, searches for images that fool the detector, but it needs gradient access to the generator's weights (Where Detectors Fail), which a commercial API does not provide. When a new closed generator appears, a defender usually has only a few dozen of its outputs.

We propose to train a language model to write short, executable detection programs from those few samples. Given a handful of images from an unseen generator and a set of real images, the model writes a program: a spectral filter, a neighboring-pixel residual, an autoencoder reconstruction error, a perturbation-stability test on foundation-model features, or some new combination of these. It runs the program in a sandbox, reads diagnostics on a development split, and revises the code. Verified programs go into a growing library. When a new input arrives, the system first scores stored programs by running them, not by embedding similarity, so it can tell when its library does not cover a generator and only then writes a new program. The model is post-trained in two stages. Supervised fine-tuning on successful probe-writing trajectories comes first. Reinforcement learning follows, rewarded by how well the program separates real from fake on held-out images it never saw during refinement.

Core Claim

In the regime of few samples, black-box access, and an unseen generator family, a program-writing policy trained with execution feedback produces detectors that generalize better than two alternatives given the same samples: fitting a classifier on foundation-model features, or combining a fixed set of known forensic operators. The claimed advantages are three. Programs are low-capacity, which limits overfitting to a few dozen images. They can express structural hypotheses such as "this decoder leaves periodic energy at a new frequency" that a linear probe cannot state. They can be checked and stored as reusable knowledge. A second claim concerns knowledge coverage. Retrieval by execution score gives a calibrated signal of missing coverage, because every stored program fails to separate the new samples. Embedding-based retrieval does not give this signal: it always returns a nearest neighbor, even when that neighbor is irrelevant.

Relation to Prior Work

Three lines of work motivate the choice of what the programs should contain. NPR shows that differences between adjacent pixels expose upsampling traces. Aligned Datasets shows that reconstructions through the generator's autoencoder produce a shared fingerprint. RA-Det shows that features of synthetic images in CLIP- or DINO-style models drift more under structured perturbation than features of real images. Each is a fixed, hand-designed operator. We treat them as building blocks the policy can call, rewrite, and compose, rather than as final detectors.

PROBE (Where Detectors Fail) motivates the adaptive framing, since it finds that passively collecting more data does not close the generalization gap. It adapts through white-box search over the generator; we adapt through code search using only sampled outputs.

Two papers are the closest work, and both apply reinforcement learning to detection. PRPO and BusterX++ train multimodal language models to output labels and paragraphs of explanation. Their action is text, which PRPO shows can invent forensic evidence that is not there. Our action is a program whose claim is checked by running it.

On the code side, ARCS supplies the method template: retrieve, synthesize, execute, repair within a fixed iteration budget, and keep the best version so far. ARCS runs frozen models on standard coding benchmarks. We post-train the policy and replace unit tests with a statistical test on held-out images. Yan et al. constrain the design: prompting-only refinement loops plateau after a few rounds, which is the direct motivation for learning the refinement behavior with reinforcement learning. The skill libraries of LearnAct and OS-Copilot supply the idea of a self-growing library of verified code.

On knowledge coverage, Re³Cap and RA-TTA both report that their gains disappear when the reference store lacks relevant items. We test whether synthesizing a new entry, gated by an execution-based signal that coverage is missing, recovers what those systems lose.

The file also shows a disagreement on whether supervised fine-tuning is needed first. Wen et al. argue that it prevents the policy from collapsing during reinforcement learning. BusterX++ reports reinforcement learning without it beating a fine-tuned model (77.4% vs 72.9% on GenBuster++). We test both orderings rather than assume either.

Research Program

We build a sandbox that exposes image arrays, standard signal-processing libraries, and read-only access to frozen feature extractors (DINOv2, CLIP) and public autoencoders. The sandbox has no network access and no access to file metadata. Srivastava et al. found that iterative prompting loops can amplify security vulnerabilities, so code generated by the model runs isolated.

Each training episode has four parts. First, a generator is sampled from the training families. Second, we draw a support set of $k$ fake and $k$ real images, a development split whose results the policy may read, and a test split that it never sees. Third, the policy writes and revises a program that outputs one score per image, within a budget of $B$ execution rounds. Fourth, the episode is scored. All images are resized and re-encoded to a common format before the program sees them. Without this step, a program could separate real from fake by file format, and GenImage-style data carries such format biases.

The reward for a final program $p$ is

$$R(p) = \frac{1}{|T|}\sum_{t \in T} \mathrm{AUROC}_{\text{test}}\big(p \circ t\big) - \lambda\, c(p).$$

Here $T$ is a set of post-processing transforms (identity, JPEG at several quality levels, blur, resize), and $\mathrm{AUROC}_{\text{test}}(p \circ t)$ is the area under the ROC curve of the program's scores on transformed test images. The term $c(p)$ is the program's runtime cost and $\lambda$ weights it. Averaging over $T$ penalizes programs that rely on fragile high-frequency cues, which anti-forensic methods such as Untraceable DeepFakes remove on purpose.

The first stage, supervised fine-tuning, uses trajectories from a large frozen model run in the ARCS style. We keep only trajectories whose final program exceeds a threshold on the test split, which is rejection sampling. The second stage runs Group Relative Policy Optimization, a reinforcement learning method that compares several sampled attempts at the same task, over several refinement turns, with $R$ as the only reward.

At deployment, the library returns the stored program with the highest separation score on the support set of the new generator. If that score falls below a threshold set on validation generators, the system declares coverage missing and synthesizes a new program. A program that verifies on held-out images is added to the library.

Generators are split by family, not by individual model. Training uses GAN families and Stable Diffusion v1.x-style latent diffusion. Evaluation uses Midjourney, DALL-E 2, FLUX.1, Midjourney v6, and the in-the-wild sets. This split puts decoder changes like FLUX's genuinely out of distribution.

Baselines

The naive reference point is a detector trained once and applied without adaptation. We use RA-Det, which the file reports at 93.47% average accuracy over 16 unseen generators when trained on ProGAN alone (RA-Det), and NPR, reported at 95.3% mean accuracy on diffusion datasets (NPR). We rerun both from released settings on our evaluation generators. Their high averages probably hide per-generator failures. The file gives no per-generator number for FLUX.1 or Midjourney v6, so we measure those ourselves, and the evaluation reports each hard generator separately. Beating these baselines means higher AUROC on the generators where they fall below 85%, not a higher average.

The strongest competitor in the few-sample regime is simple adaptation of a foundation model. We fit a logistic-regression probe, and separately a low-rank fine-tune, on DINOv2-ViT-L and CLIP features, using exactly the same $k$ support images plus the development images, with a tuning budget matched in GPU-hours to our refinement budget. No result for few-shot linear probes on these generators was found on Lacuna. The closest reported figure is 83.37% accuracy from only 1,000 aligned pairs (Aligned Datasets), which we reproduce at matching scale. PROBE's 93.9% average balanced accuracy on DINOv2-ViT-L (Where Detectors Fail) serves as a white-box upper reference, not a like-for-like competitor, because it uses generator gradients that our setting withholds.

The ablation that isolates program synthesis is a fixed operator library with a learned combiner. The library holds the NPR residual, spectral statistics, autoencoder reconstruction error, and RA-Det-style feature drift, and a small classifier combines them on the same support set. A second ablation keeps our full pipeline but removes the policy's ability to write new code, leaving only retrieval over the library. This measures what synthesis adds once coverage is missing.

Three training-regime baselines separate what each training stage contributes. The first is the frozen large model in the ARCS style with the same budget $B$. The second is the model after supervised fine-tuning only. The third is reinforcement learning without supervised fine-tuning, which tests the BusterX++ claim. Finally, BusterX++, a language-output model trained with reinforcement learning, is run on the image portion of GenBuster++, where its reported overall accuracy is 77.5%, to compare explanations written as text against explanations written as programs.

Why not just…?

Why not just fit a linear probe on DINOv2 features from the same few samples? This is the most serious threat, and we have no reported number showing it fails in this regime. Understanding Why Foundation Models Work suggests these features already carry much of the relevant signal. The case for programs rests on three bets: overfitting at $k \le 32$, fragility under JPEG compression and blur, and the fact that programs may call DINOv2 themselves and so can match the probe at worst. The experiment varies $k$ from 8 to 512. If the probe matches our programs at every $k$ and under every transform, the synthesis claim is refuted and we report that result.

Why not just combine known forensic operators instead of writing new code? If new generators leave traces that existing operators already capture, a combiner over a fixed library is cheaper and just as good. The fixed-library ablation tests exactly this. The expected gap is concentrated on generators with new decoders, such as FLUX.1, where the authors of Aligned Datasets acknowledge that reconstruction-based fingerprints fail. If there is no gap on those generators, synthesis adds nothing.

Why not just prompt a strong frozen model in a refinement loop, as ARCS does? Yan et al. report that such loops plateau quickly. Forensic diagnostics are also far less informative than a compiler traceback. We plot test AUROC against refinement round for the frozen model and for our trained policy. If the frozen loop reaches the same final AUROC, the reinforcement learning stage is not earning its cost.

Why not just scale the training set to many generators? PROBE reports that data scale alone does not close the gap. More importantly, a static corpus cannot contain a generator released after it was built. We test this directly by training the naive baselines on every training family pooled together and measuring accuracy on the held-out families.

Proposed Experiments

The first decisive check is the $k$-sweep on held-out families. We measure test AUROC, averaged over transforms, for our method, the linear probe, and the fixed library at $k \in \{8, 32, 128, 512\}$ on Midjourney, FLUX.1, Midjourney v6, and the Chameleon, WildRF, and SynthWildX sets. The core claim requires our method to win at small $k$ on the families with new decoders. The advantage may narrow at large $k$, and a gap that closes is acceptable.

The second check tests the coverage signal. We measure whether "no stored program separates the support set" predicts that retrieval-only accuracy will be low, as the area under the ROC curve of that signal across held-out generators. We compare it with a signal based on embedding-similarity retrieval.

The third check concerns training order. We compare supervised-then-reinforcement training with reinforcement learning alone, measuring final reward, the rate of programs that fail to run, and the diversity of synthesized programs, to settle the disagreement between Wen et al. and BusterX++ for this code-writing action space.

The fourth check tests whether programs overfit the development split. We measure the gap between development and test AUROC across refinement rounds, and we inspect by hand 100 synthesized programs for shortcuts such as image statistics tied to file format.

The fifth check is robustness. We evaluate against evasion by applying fingerprint-removal attacks (Untraceable DeepFakes, TraceEvader) to fakes from the held-out generators, then re-running synthesis on the attacked samples.