Fault-Injected Twins: Measuring Where Image Quality Metrics Stop Seeing Defects

Image generators fail quietly. A diffusion model, which builds an image by gradually removing noise from a random starting point, can run to completion, return a valid image, and still produce a sixth finger, a melted face, or a smeared object. Software reliability research has a name for this, "silent degradation," and a standard tool for studying it, automated fault injection: deliberately breaking parts of a system on purpose so you can watch how failure spreads and whether monitors catch it. We propose bringing that tool inside generative image pipelines. We would corrupt specific internal signals at controlled strengths, namely attention to particular prompt words, regions of the intermediate latent image, individual denoising steps, and the decoder's numerical precision. Every faulty image would be produced alongside a "twin" made from the same random seed with no fault. Paired with human judgments, these twins let us measure, for each fault type, the exact strength at which people first notice something wrong and the strength at which each quality metric first reacts. The difference between those two thresholds is a direct, quantitative measure of how badly a metric misses what people see.

The practical stake is whether quality metrics can serve as automatic gates that stop broken images before they reach users. In text pipelines, Compound AI System Reliability found that 51% of production incidents were silent degradations that took an average of 4.2 days to detect, and that output quality gates caught 73% of them. Nobody knows the equivalent number for image pipelines. The evidence we have is discouraging. AGIDefect-4K reports that over 85% of generation defects cover less than 5% of the image, and that zero-shot GPT-4o and Gemini reach detection AUCs of only 0.29–0.38. Our claim is that internal fault injection yields defects that look like real generator failures, with known cause and strength. That makes it possible to audit metrics and to train better detectors, and it does both better than pixel-level synthetic damage or passively collected defect images.

Core Claim

Faults injected inside the generator produce defects that lie on the generator's own output distribution. Their strength can be dialed, and each comes with a clean twin. Three things follow, and each can be tested. First, for every metric and fault type we can estimate a human visibility threshold $d_h$ and a metric response threshold $d_m$. A metric is misaligned on a fault type to the extent that $d_m - d_h$ is positive, meaning people see the defect before the metric reacts. Second, detectors trained on fault-injected images should transfer to naturally occurring defects better than detectors trained on pixel-space corruptions of equal size and cost. Third, the twins turn the operational question, "would this gate have caught it?", into a measurable catch rate at a fixed false-alarm rate.

The first claim is mostly measurement. The second is the risky, falsifiable part, and it is where the idea could fail.

Relation to Prior Work

Two lines of work motivate the proposal. Fault-injection research established boundary interception as a method: RCAEval injects 11 fault types into microservices to produce labeled diagnostic benchmarks, and Compound AI System Reliability injects 23 fault modes at the interfaces between components of text and retrieval pipelines. Instruction Bleed adds a design we borrow directly. It applies paired perturbations through separate channels and finds that content injection shifts scores (Cohen's d = 0.63) without ever flipping a decision, which is the signature of silent degradation. None of these touch images or perceptual metrics. Our injection machinery borrows from neural-network fault injectors such as MRFI and enpheeph, which flip bits in weights and activations to study hardware reliability. We aim the same mechanism at perceptual outcomes rather than accuracy.

On the perception side, Human Preference Score and AGIDefect-4K document that standard metrics are blind to anatomical defects. Both rely on static collections of generator outputs, so neither can say what caused a defect or how strong it was. Towards Explainable Partial-AIGC Image Quality Assessment finds a "weakest link" collapse, in which perceived quality drops sharply once either harmony or naturalness falls below a threshold. That is exactly the kind of threshold effect our dose-response curves measure directly. One paper constrains the design. Attacking Perceptual Similarity Metrics shows that tiny adversarial input perturbations flip the rankings of LPIPS, DISTS, and others, and that the metrics most aligned with humans are the most fragile. We do not study worst-case attacks. Our faults model realistic operational failures, not optimized perturbations. That paper does tell us to report robustness separately from clean-data alignment, and to include a robustly trained metric, BiRQA, which itself admits weakness on geometric distortions of the kind our attention and latent faults are likely to cause.

Research Program

We would build a fault harness around an open text-to-image diffusion pipeline. The Stable Diffusion family is the natural choice, since Human Preference Score builds on it. The harness has hooks at five boundaries, each with a continuous strength parameter. The first is the text-conditioning embedding, where we add noise or swap individual token embeddings. The second is cross-attention, the layers that tie image regions to prompt words; here we drop heads or attenuate attention to selected tokens such as "hand." The third is a spatially masked region of the intermediate latent image, which we corrupt at a chosen denoising step. The fourth is the step schedule, where we skip or repeat steps. The fifth is the decoder that maps latents to pixels, where we lower numerical precision or flip bits.

Prompts are drawn from the Human Preference Dataset. Each fault type, strength, and seed combination produces a faulty image and its clean twin.

Humans then rate a stratified sample of these images in two protocols. One is a forced choice between twins, which gives the visibility threshold. The other is a single-image judgment: does this image contain a defect, and if so, where? We need the second because deployed gates never see a twin. The location masks people draw are necessary because an injected fault does not guarantee a visible defect in the injected region. We treat the human mask as ground truth, not the injection site.

From these data we fit, for each fault type, the probability $p_h(d)$ that a person notices the defect at strength $d$, and the equivalent detection probability $p_m(d)$ for each metric at a threshold calibrated to a 5% false-alarm rate on clean images. Here $d_h$ and $d_m$ are the strengths at which these curves reach 50%.

We audit two families of metrics. The first is full-reference metrics, which score an image against the clean twin: PSNR, SSIM, LPIPS (The Unreasonable Effectiveness of Deep Features as a Perceptual Metric), DISTS, and BiRQA. The second is no-reference metrics usable in deployment: HPS, AGIDA from AGIDefect-4K, the EPAIQA model, and zero-shot vision-language models.

We then train defect detectors on three kinds of data. The first is fault-injected images. The second is pixel-space corruptions with matched size, location, and count, made by pasting local blur, warps, or patches from other images into clean images. The third is the AGIDefect-4K training split. All detectors are evaluated on held-out natural defects.

Baselines

The naive reference point is the pipeline's existing health check, which only notices whether the run finished. By construction it catches nothing silent. It is joined by PSNR and SSIM computed against the clean twin. These are the cheapest possible gates, and they define the floor any learned metric must clear. SSIM's roughly 63% forced-choice agreement on BAPPS, against a 73.9% human ceiling (The Unreasonable Effectiveness of Deep Features as a Perceptual Metric), suggests it will lag. We will report its visibility gap rather than assume it.

The strongest existing detector is AGIDA, a Qwen3-VL model with a segmentation encoder trained on AGIDefect-4K. The original paper reports detection AUC 0.65 and quality SRCC 0.800 on AGIDefect-4K (AGIDefect-4K). It is the strongest competitor because it is the only detector in the file trained specifically on localized generation defects. For a fair comparison we keep its backbone and training recipe fixed and only change the training data: AGIDefect-4K alone, AGIDefect-4K plus fault-injected images, and AGIDefect-4K plus pixel-corrupted images. The added sets are equal in size, and every variant gets the same number of steps and the same hyperparameter search. We count it as beaten if adding fault-injected data raises AUC on the AGIDefect-4K test split above the reproduced AGIDA number by more than the variation across seeds.

For full-reference alignment we compare against BiRQA. It reports SROCC 0.988 on LIVE and 0.979 on CSIQ, and keeps 0.60–0.84 SROCC on KADID-10k under white-box attack where undefended models drop to 0.30–0.57 (BiRQA). It is the strongest metric that combines robustness with alignment. Here we do not need to beat it. The question is whether its gap $d_m - d_h$ on our faults is larger than its benchmark numbers would predict.

The key ablation isolates what internal injection contributes. It is the pixel-corruption training set, matched to the fault-injected set in defect area, location distribution, image count, and annotation budget.

Why not just…?

Why not just use AGIDefect-4K's natural defects? They are the real target distribution, which is why we evaluate on them. But they number 3,000 defective images with no information about cause or strength, so they cannot produce visibility thresholds or say which pipeline component failed. Our transfer experiment directly tests whether injected data adds anything beyond them. If adding it does not raise AUC above the AGIDefect-only detector, the training half of this proposal fails. The audit half survives regardless.

Why not just paint synthetic defects in pixel space? Pixel-space corruption is much cheaper, and it is the most serious threat to the proposal. Our argument is that defects generated inside the model share the generator's textures and lighting, while pasted damage leaves seams that detectors learn as a shortcut. That argument is a hypothesis, not evidence. The matched ablation settles it. If pixel-corrupted training data matches fault-injected data on natural-defect AUC, we would recommend pixel corruption for training and keep the twin harness only for auditing.

Why not just ask a strong multimodal model to judge each image? Zero-shot GPT-4o and Gemini reach AUCs of 0.29–0.38 on AGIDefect-4K, below chance, which the authors attribute to positive bias (AGIDefect-4K). We still include them in the audit, because the dose-response curves show whether their blindness is uniform or limited to faint defects. That distinction matters for using them as gates.

Why not just reuse adversarial attacks on metrics? Attacking Perceptual Similarity Metrics already measures worst-case fragility. Worst-case results do not tell an operator how often realistic failures slip through. Our faults are not optimized against any metric. If metrics turn out well aligned on them, that is a useful negative result: adversarial fragility would then overstate the operational risk.

Proposed Experiments

The first experiment estimates visibility gaps. For each fault type we estimate $d_h$ from human forced-choice and single-image ratings, and $d_m$ for each audited metric at a 5% false-alarm rate. We report the gaps with bootstrap confidence intervals, following the paired-statistics practice of Instruction Bleed. We expect, without assuming it, that full-reference deep metrics will react before people do on decoder and precision faults, and after people do on attention faults that produce anatomical errors. That would match the texture-versus-structure blindness reported by Human Preference Score.

The second experiment is the transfer test and is the decisive check. We train detectors on fault-injected, pixel-corrupted, and AGIDefect-only data and evaluate AUC and localization quality on the AGIDefect-4K test split. As a secondary check we evaluate on EPAIQA-15K edits, measuring SRCC against overall quality. That paper reports 0.631 for its own model (Towards Explainable Partial-AIGC Image Quality Assessment).

The third experiment simulates a gate. We place the best no-reference detector after the generator in a serving loop with faults injected at a realistic rate. We report the fraction of human-visible degradations caught at a fixed false-alarm rate, the added latency per image, and the error introduced if caught images are simply regenerated. Text pipelines catch 73% with output gates (Compound AI System Reliability). That figure gives context only and is not a target, since no catch rate for visual pipelines has been reported on Lacuna.

The proposal succeeds if the gaps are large and differ systematically across fault types, and if internal injection beats matched pixel corruption on natural defects. It fails cleanly if pixel corruption does just as well.