Counterfactual Averaging: Extracting Static Watermarks With No Clean Control Group

Generative AI models are increasingly shipped with watermarks—hidden statistical patterns stamped into every image, audio clip, or paragraph the model produces, so that a detector can later certify the content as machine-made. Many deployed schemes are static: the same pattern, or the same vocabulary bias, is imprinted on every output regardless of content. This persistence is a known weakness. If an adversary collects a few hundred generations, the natural content blurs together under averaging while the fixed pattern reinforces, lifting a high-fidelity copy of the watermark that can be subtracted off (scrubbing) or added to human-made media (forgery). But the demonstrated black-box version of this attack has a hidden dependency that mirrors a classic problem in medicine: it needs a control group. Can Simple Averaging Defeat Modern Watermarks? had to subtract an external pool of clean natural images (ImageNet) from the average of watermarked outputs, because natural images do not have zero mean—their residual luminance and texture statistics otherwise contaminate the estimate. In domains where no representative clean distribution exists (stylized synthetic art, rare prompt spaces, medical imagery), the attack silently breaks.

This is precisely the "single-arm trial" problem: we observe only treated units (watermarked generations) and never the counterfactual (what the same model would have produced unwatermarked). Clinical research has developed an arsenal for exactly this situation—synthetic controls that impute missing counterfactuals from weighted donor pools, and placebo tests that expose when such imputation is untrustworthy. Nobody has brought that machinery to watermark security. This proposal does.

Core Claim

A static watermark can be estimated from watermarked outputs alone—with no external clean control set—if we treat watermark extraction as a causal estimation problem. Write each watermarked image as $I_i = C_i + W$, where $C_i$ is the unwatermarked counterfactual content and $W$ is the fixed watermark (the "treatment effect"). Naive averaging fails because

$$\hat{W}_{naive} = \frac{1}{N}\sum_{i=1}^N I_i = W + \bar{C}_N, \qquad \bar{C}_N \not\to 0,$$

since $\bar{C}_N$, the mean of natural content, retains non-zero structure—this is exactly why Yang et al. were forced to import an ImageNet control. Our estimator instead imputes each $C_i$:

$$\hat{W} = \frac{1}{N}\sum_{i=1}^N \left(I_i - \sum_j w_{ij} D_j\right),$$

where $D_j$ are donor generations produced by an open, unwatermarked model conditioned on the same prompts, and the weights $w_{ij}$ form a synthetic control: a regularized combination of donors matched to image $i$ on covariates that are measurable on the treated output and provably (approximately) unaffected by the watermark—CLIP embeddings and low-frequency image statistics, since additive watermarks live at low amplitude and high frequency. This borrows the convex-combination matching of Robust Synthetic Control and the time-weighted donor logic of Time-Aware Synthetic Control, both of which construct counterfactuals for a treated unit from untreated donor units, and adapts them to a setting where the "pre-treatment covariates" are watermark-invariant features of the output itself.

A second, equally important claim concerns validation. Synthetic-control methodology comes with built-in honesty checks: placebo tests that run the estimator on units known to be untreated. Applied here, the estimator is run on generations from an unwatermarked model; the resulting residual defines a null distribution, turning "is this a watermark?" into a hypothesis test with a controlled false-positive rate. This directly addresses a hole in current watermark evaluation, where degradation measurements conflate adversarial damage with natural generation drift—just as single-arm clinical trials conflate drug effect with natural disease fluctuation, as documented in psoriatic arthritis by Sfikakis et al. and in diabetes by Masuda et al..

What Exists and What This Adds

Three lines of work bracket this proposal without covering it. The averaging attack of Can Simple Averaging Defeat Modern Watermarks? supplies the threat model and the empirical target—Tree-Ring detection AUC collapsed from 1.0 to roughly 0.24 under their attack—but its black-box variant requires an unpaired clean pool, and the paper leaves open what happens when that pool is biased or absent. The BOWS contest postmortem by Le Guelvouit et al. shows the extreme case: their "Bad" attack worked only because the organizers had accidentally published an exact unwatermarked control image (the "church"), achieving 37.35 dB PSNR by ground-truth subtraction—opportunistic, with no systematic method for when no such control exists anywhere, which is the universal situation for prompt-native generations. In text, B⁴ distills a proxy of the watermarked distribution but represents the unwatermarked counterfactual with an off-the-shelf paraphraser—an unvalidated proxy control whose stylistic confounding the authors themselves do not account for. On the causal side, MTAL in basket trials generates missing counterfactuals for single-arm clinical cohorts via multi-task adversarial training, but has never been applied outside oncology data. Our contribution is the intersection none of these occupy: a single-arm estimator of static watermark signals, with synthetic-control imputation of the missing clean distribution and placebo-based attribution, validated on watermarking benchmarks where the ground truth watermark is known.

The proposal also engages a live contradiction in the literature. Invisible Image Watermarks Are Provably Removable Using Generative AI reports that Tree-Ring retains nearly 100% detection under regeneration attacks, concluding semantic watermarks are robust; the averaging paper reaches the opposite conclusion, showing Tree-Ring's initial-noise modification propagates a static low-level ripple that averaging strips (AUC 1.0 to ~0.24). The disagreement is a baseline-structure disagreement: single-sample reconstruction versus multi-sample statistics. Our framework adjudicates it, because it can test whether the Tree-Ring residual behaves as an extractable invariant bias (recoverable by our estimator) or as model-intrinsic content (absorbed into the counterfactual imputation and thus not extractable).

Research Program

The program has three parts. First, a formal analysis: under the Simple Linear Assumption, $W$ is identified by our estimator if and only if the prompt-conditional mean gap between the victim model and the donor pool is watermark-free—directly analogous to the parallel-trends condition in synthetic control. We will characterize when this fails (a victim model whose style itself is inseparable from the watermark signature) and derive the bias of the estimator as a function of donor–victim distribution mismatch, using the moment-matching machinery of Robust Synthetic Control and, where privacy constraints on donor data arise, the noise guarantees of Differentially Private Synthetic Control. Second, the estimator itself: donor generations from open unwatermarked Stable Diffusion variants, convex or elastic-net weights fit per image on watermark-invariant covariates, then averaged into $\hat{W}$, followed by scrubbing ($I_i - \hat{W}$) and spoofing ($X_{clean} + \hat{W}$). Third, the validation layer: placebo runs on unwatermarked victims (inexact placebos use donors from a different model family), yielding a false-attribution threshold; this is the piece current watermark evaluation lacks, and it connects to the evaluation-confounding gap identified in CEFW, where watermark decay under attack is measured without any counterfactual baseline for natural generation drift.

Proposed Experiments

Image experiments use Stable Diffusion v1.5/v2.1 as victim, prompted from Stable Diffusion Prompts and DiffusionDB (the same corpora used by the averaging paper), with $N \in \{50, 100, 500\}$ generations per key. Targets are Tree-Ring and content-agnostic additive schemes; because we embed the watermarks ourselves, we have ground-truth $W$ and can report residual error $\|\hat{W} - W\|$ directly, alongside the attack metrics: post-scrub detection AUC, spoof acceptance rate on clean COCO2017 images, and PSNR of scrubbed outputs. Domain shift is the crux: on stylized or rare prompt subsets we compare our estimator against the ImageNet-control baseline, predicting the external control degrades (its donor pool is mismatched) while the prompt-matched synthetic control does not; the placebo test quantifies each method's false-extraction rate. Two validation anchors close the loop. On the BOWS "church" image, we estimate the watermark without using the organizers' published original and compare the scrubbed PSNR against the 37.35 dB ground-truth-subtraction result reported by Le Guelvouit et al.. On Tree-Ring, we test whether the ripple is extractable, adjudicating the contradiction with the regeneration-attack paper. Extensions follow the same template in audio (AudioSeal on the Common Voice zh-CN subset, where the averaging paper reports detection falling from 1.0 to roughly 0.75 under their attack with a clean control) and text (the B⁴ setup on C4-Realnewslike with Llama-2-7B victims, replacing the paraphraser proxy with a donor-model counterfactual and measuring whether scrubbing fidelity improves; B⁴ reports up to a 68.13% improvement in attack success over DIPPER-style paraphrasers, which we take as the bar the counterfactual control must clear).

Baselines

The naive reference point is zero-mean averaging: estimate $\hat{W}$ as the plain mean of $N$ watermarked images, assuming content averages to nothing. It requires no donor data and no control, and it is exactly the shortcut the averaging paper found untenable—natural images carry non-zero mean statistics, leaving residual content blurred into $\hat{W}$, which degrades scrubbed-image quality and produces artifacts in forgeries. We run it at the same $N$, same prompts, and same detectors so its failure is measured, not asserted.

The strongest existing competitor is the black-box averaging attack of Can Simple Averaging Defeat Modern Watermarks?, which subtracts an unpaired ImageNet control mean from the watermarked mean. This is the method that set the benchmark result we must reckon with: Tree-Ring detection AUC fell from 1.0 to approximately 0.24, while content-adaptive schemes (RivaGAN, HiDDeN) held detection above 0.95. We run it under identical conditions—same victim model, prompts, sample sizes, detectors, and compute—first on natural-image-like prompts (where it should be strong, and where matching its AUC drop without any external clean set is our minimum claim), then on the shifted-domain subsets (where we predict its control becomes a liability). Beating it means achieving a comparable or larger AUC reduction and spoof success in the shifted domains, with scrubbed-image PSNR at or above its level, while using no clean data it does not also get.

The strongest single-sample competitor is the regeneration attack of Invisible Image Watermarks Are Provably Removable Using Generative AI, available as XuandongZhao/WatermarkAttacker. It removes 99% of DwtDctSvd, RivaGAN, and SSL watermarks at PSNR above 30 dB, but the same paper reports Tree-Ring retains nearly 100% detection—so on our primary target it is expected to lose, and we say so in advance rather than argue around it. The fair comparison fixes a total generation-query budget: regeneration spends one pass per image, our method spends $N$ queries once and then scrubs every future output for free; we report the full budget-versus-AUC curve for both. Finally, an ablation isolates the synthetic-control ingredient: the same donor pool and the same averaging, but with randomly assigned donors instead of covariate-matched weights. If matching does not matter, the causal framing is decoration; this ablation is what proves it is not.

Why not just…?

Why not just scrape a clean control set from the web? This is the status quo and, in natural-image domains, it works—that is the honest starting point. The averaging paper's ImageNet control drove Tree-Ring AUC to ~0.24. The claim of this proposal is not that external controls are useless, but that they are an unvalidated proxy whose failure mode is silent: when the victim model's output domain diverges from the scraped pool, the estimated $\hat{W}$ absorbs domain bias, and nothing in the current pipeline detects it. Our shifted-domain experiments measure exactly this degradation, and our placebo test converts it into a visible, quantified false-extraction rate. If the ImageNet-control baseline matches our estimator even on shifted domains, the proposal's central claim fails and we will report that.

Why not just assume the content averages to zero? The averaging paper already showed it does not: natural images retain non-zero mean luminance and chrominance, which is why its authors were forced to import a control in the first place. The zero-mean shortcut is our naive baseline and will be run; the expected signature of its failure is a $\hat{W}$ with visible low-frequency content residue, degraded scrubbed-image PSNR, and reduced spoof acceptance. It is cheap, so it must be beaten on quality of the extracted fingerprint, not just on AUC.

Why not just use regeneration attacks, since the code is public? Because they attack a different vulnerability. Regeneration is a per-sample purification: it removed 98–99% of adaptive pixel watermarks like RivaGAN at PSNR above 30 dB, but left Tree-Ring at nearly 100% detection, per the same paper. Our target is precisely the multi-sample invariant footprint that regeneration cannot touch, and the Tree-Ring adjudication experiment will show whether the ripple is in the extractable class. The two attacks are complementary, and a defender must survive both; on adaptive watermarks we expect regeneration to remain the better attack, and we frame it that way rather than claiming universal superiority.

Why not just switch defenders to content-adaptive watermarks and declare the problem moot? Adaptive schemes survived averaging (detection above 0.95 for RivaGAN and HiDDeN), but regeneration removed 98–99% of RivaGAN at PSNR above 30 dB, and CEFW documents that fully dynamic text watermarks pay in perplexity, generation cost, and edit-robustness—Provable Robust Watermarking for AI-Generated Text shows the static unigram design keeps ~0.86 TPR under paraphrasing where context-dependent KGW drops to ~0.56. The robustness–security trade-off means static designs will persist, and every watermark, static or adaptive, still needs the control-group-free evaluation machinery this proposal builds. Architecture-level defenses like MOLM (98% bit accuracy at FID degradation below 1.5, and resilient to averaging) raise the bar for extraction, and we will include it as a target to see whether model-internal routing genuinely defeats counterfactual averaging or only the additive variant.

Decisive Checks

The proposal lives or dies on four results. The shifted-domain comparison: our estimator must beat the ImageNet-control averaging attack where the control pool is mismatched, or the contribution collapses to a re-derivation. The placebo: run on an unwatermarked victim, the estimator must produce a residual statistically indistinguishable from its null distribution; a non-trivial false-extraction rate would mean we are extracting model style, not watermark. The Tree-Ring adjudication: a clean answer, whichever way it goes, resolves a published contradiction. And the honesty check throughout: wherever a simpler baseline—the scraped control, the zero-mean average, the public regeneration code—matches us at equal budget, we report it as the better method and keep only the evaluation methodology, which no existing work provides.