Calibrated Inpainting Residuals for Anomaly Detection

A common way to spot anomalies with a generative model works like this. Hide part of an image, ask a model trained only on normal data to fill the hidden part back in, and measure how different the fill is from what was actually there. A large difference suggests the hidden content was abnormal. The difference, however, has two sources. One is abnormal content. The other is the masking itself: when a large or information-rich region is hidden, many completions are plausible, so even a perfectly normal image comes back looking different. Current methods mix the two sources into one raw residual. They control the second only by hand-picking a mask size or noise level. If the mask is too large, normal images fail to reconstruct. If it is too small, anomalies leak through from the surrounding context. The same mixing appears in explanation methods that in-fill masked features, and it parallels a known failure of likelihoods: simple, low-complexity inputs look "normal" to a density model for reasons unrelated to whether they belong to the training data.

We propose to separate the two sources inside the reconstruction model itself. The model is asked for several completions of each hidden region, as current diffusion-inpainting detectors already do before throwing the extra samples away. We then test whether the true content looks like one more draw from that set of completions. The result is a per-region rank, a p-value in the sense of conformal prediction (a distribution-free way to turn a score's rank among reference samples into a calibrated probability). Suppose the image is normal and the inpainter's spread of completions is roughly right. Then the true content is interchangeable with the samples, and its rank is approximately uniform. This holds regardless of mask size or image complexity. The perturbation's own effect is thereby measured and removed per image and per region, not tuned away globally.

Core Claim

Reconstruction residuals suffer from a confound created by the perturbation: how much the hidden content is left undetermined by its context. This confound inflates scores on complex normal images and deflates them on simple ones. Replacing the raw residual with the rank of the observed region among the model's own completions yields three properties. The score's behaviour on normal data stays stable across mask sizes and across image-complexity strata. Sample-level out-of-distribution detection is at least as good as raw inpainting residuals. Pixel-level localization comes without per-dataset blur or prototype tuning. The key ingredient is small and easy to isolate: the same completions, scored by rank instead of by median distance.

Relation to Prior Work

The method builds directly on Unsupervised Out-of-Distribution Detection with Diffusion Inpainting. That paper introduced Lift, Map, Detect (LMD): hide pixels with a checkerboard mask, in-fill them with a diffusion model trained on in-distribution data, and score by perceptual (LPIPS) distance, taking the median over several stochastic runs. LMD motivates the proposal, since it shows inpainting sidesteps the likelihood failure. It also leaves the gap we target: its mask size is heuristic, and it uses the spread of its samples only to stabilise a median, never as a reference. We keep its inpainter and samples and change only the score.

Two lines of anomaly-detection work fight the "identical shortcut", where a high-capacity reconstructor simply copies anomalies. A Unified Model for Multi-class Anomaly Detection (UniAD) blocks self-copying with neighbour-masked attention and feature jittering. Vague Prototype-Oriented Diffusion Model for Multi-Class Anomaly Detection (VPDM) low-pass filters features and maps them to normal prototypes before diffusion refinement. Both control how much information survives the perturbation through a fixed architectural or filtering choice. VPDM's filtering needs careful tuning because excessive blurring removes legitimate normal detail. We do not try to find the right amount of information removal. We measure, per region, what the chosen removal does to normal data. These methods constrain our design, because they show copying is real. They are also our strongest localization competitors.

The likelihood-paradox literature supplies the analogy and a baseline. Do Deep Generative Models Know What They Don't Know? showed that density models rate SVHN above their own CIFAR-10 training data. Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models traced this to input complexity and corrected it by subtracting a compression-based complexity estimate. A Geometric Explanation of the Likelihood OOD Detection Paradox explained it by the lower intrinsic dimension of simple data, and proposed pairing likelihood with a local intrinsic dimension (LID) threshold. Our rank test performs a complexity correction of the same spirit. The difference is that the reference is conditional on the image's own visible context, rather than a global compressor or a global dimension threshold. Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder also compares an input against what the model can do for it, but through latent optimisation of one VAE rather than a sample-based rank. EigenScore: OOD Detection using Posterior Covariance in Diffusion Models is the closest in spirit: it uses the diffusion posterior's covariance as the score. We use the spread only as the yardstick against which the observed content is ranked. An ablation below tests whether spread alone suffices.

The explanation literature frames the confound. Explaining Image Classifiers by Counterfactual Generation and A Rate-Distortion Framework for Explaining Black-box Model Decisions in-fill masked regions generatively so that classifiers are not judged on gray boxes. The file records an open "plausibility versus informativeness" problem: such in-fills can borrow context and hide the removed information. Benchmarking Deep Learning Interpretability in Time Series Predictions and Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration show that the perturbation itself, not information removal, often drives the measured effect. Our score is precisely a per-region test of whether a fill is an ordinary draw from the prior given context. It therefore doubles as a diagnostic for explanation in-fills, which we treat as a secondary application.

Research Program

Let $M$ be a mask hiding region $r$ of image $x$. Let $\hat x^{(1)},\dots,\hat x^{(K)}$ be $K$ completions from an inpainter trained on normal data. For any candidate content $y$ of region $r$, let $\bar d(y)$ be its average perceptual distance to the completions, excluding itself. The region score is the conformal rank

$$ p_r(x) = \frac{1}{K+1}\Big(1 + \sum_{k=1}^{K} \mathbb{1}\big[\bar d(\hat x^{(k)}) \ge \bar d(x_r)\big]\Big), $$

where $x_r$ is the observed content of region $r$. Small $p_r$ means the observed content is farther from the completions than the completions are from each other. Since $p_r \ge 1/(K+1)$, one mask gives coarse resolution. We therefore also compute the continuous ratio $\bar d(x_r) / \mathrm{median}_k\, \bar d(\hat x^{(k)})$. Region scores are combined across a family of masks into a pixel map, by averaging the per-pixel ratio over the masks that cover each pixel, and into an image score through the minimum p-value. Its threshold is set on held-out normal images, because overlapping masks are not independent.

The work has three parts. First, a calibration study. On held-out normal images, we measure how the null distribution of raw residuals versus rank scores shifts as mask ratio varies. We also measure it across image-complexity strata, defined by PNG compressed length as in the complexity-correction paper. A toy Gaussian-mixture setting with an exact posterior verifies that the rank is uniform when the posterior is correct. We then measure how miscalibrated, meaning over- or under-dispersed, a real diffusion inpainter's completions are. Second, detection and localization on standard benchmarks with the inpainter held fixed. Third, the explanation diagnostic. In FIDO-style in-fill explanations, we compute the rank of the in-fill used by the explainer, and check whether low-rank fills coincide with regions where deletion curves disagree between blur and generative in-fill.

The main risk is posterior miscalibration. A mode-collapsed inpainter produces too little spread and inflates false alarms. An inpainter that is too broad on unfamiliar context misses anomalies. The calibration study measures this directly. A second risk is leakage: if a defect extends beyond the mask, completions reproduce it. Multi-scale masks address this, and we report failure rates by defect size.

Baselines

The naive reference point is LMD's own score: the median LPIPS residual, using the same diffusion inpainter, the same checkerboard masks, and the same $K$ samples. This isolates our key ingredient, since the only difference is rank versus median. LMD reports an average ROC-AUC of 0.907 across 12 dataset pairs, and 0.985 on CIFAR-100 versus SVHN, where likelihood methods reach 0.193 (Unsupervised Out-of-Distribution Detection with Diffusion Inpainting). Beating it means matching or exceeding 0.907 on the same pairs at LMD's default mask. More importantly, the AUROC range across a sweep of mask ratios should be at least halved, and the fraction of held-out normal images flagged at one fixed threshold should stay stable across the sweep.

The strongest sample-level geometric competitor is the dual-threshold likelihood-plus-LID method from A Geometric Explanation of the Likelihood OOD Detection Paradox. It reports 0.95 AUC-ROC on Fashion-MNIST versus MNIST, where plain likelihood gives 0.07. We run it with a diffusion model of matching size on identical splits, and tune its rank cutoff on the same held-out normal images we use for our threshold.

For localization, the strongest reported competitor is VPDM. It reports 98.4% image-level and 97.8% localization AUROC on unified MVTec-AD, and 94.2% AUROC on VisA (Vague Prototype-Oriented Diffusion Model for Multi-Class Anomaly Detection). UniAD reports 96.5% and 96.8% (A Unified Model for Multi-class Anomaly Detection). We quote these numbers and also rerun both with the authors' settings on our splits. Our inpainter is trained on the same normal images with a comparable GPU-hour budget. We do not expect to beat VPDM outright. The target is to come within one point of VPDM's localization AUROC while using a single mask family across all categories, with no per-dataset filter or prototype tuning. We also report per-region overlap, which, as the Cable example in MVTec AD - A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection shows, can be poor (0.05) even when AUROC is 0.86.

Two ablations complete the picture. The first scores by the completions' spread alone, in the spirit of EigenScore; no reported number was found on Lacuna for EigenScore on these pairs, so we run it ourselves. The second scores by observed distance alone, without the spread-based normalisation.

Why not just…?

Why not just tune the mask ratio on held-out normal data? This is the most direct fix, and we run it as a baseline: pick the mask ratio that maximises validation separation, which requires either a few anomalies or a proxy. The problem is that one global ratio cannot fit images whose context determines the hidden content to different degrees. The confound varies per image and per region. The calibration study tests this directly. If a tuned global ratio produces a normal-data score distribution as flat across complexity strata as the rank score does, our method adds nothing, and we will report that.

Why not just subtract a compression-based complexity estimate from the residual, as Input Complexity and Out-of-distribution Detection with Likelihood-based Generative Models did for likelihoods? This is a real threat, because it is cheap and targets the same confound. Compression, however, measures how predictable an image is under a generic code, not how predictable the hidden region is from its visible context under the normal prior. The two can diverge: a textured normal region can be highly predictable from its neighbours yet expensive to compress. We run LMD's residual minus normalised PNG length. If it matches our stability and AUROC, the simpler correction wins.

Why not just use nearest-neighbour distance on pretrained patch features? On Diffusion Modeling for Anomaly Detection concedes that k-nearest-neighbour methods often beat deep detectors at the sample level. We include a patch-feature nearest-neighbour baseline on the same backbone features used for LPIPS. The file does, however, show such feature dictionaries localize poorly: the 0.05 per-region overlap on Cable. They also produce no counterfactual, which the explanation diagnostic needs. If nearest-neighbour matches us at the sample level, our claim narrows to calibrated localization and counterfactual fills.

Why not just score by the posterior spread, as EigenScore does? Spread says how uncertain the model is, not whether the observed content falls inside that uncertainty. A complex normal region and an unfamiliar region can both have high spread. The spread-only ablation settles this. If it matches the rank score, the observed-content comparison is unnecessary.

Decisive Checks

The idea stands or falls on four measurements. The first is whether, on held-out normal images from CIFAR-10, Fashion-MNIST and MVTec-AD, the rank score's null distribution varies substantially less across mask ratios and complexity strata than LMD's raw residual. The second is whether the rank score retains LMD's out-of-distribution separation on its 12 pairs and on Fashion-MNIST versus MNIST, the hardest case for likelihoods. The third is whether it beats the complexity-subtracted residual and the spread-only score using identical samples. The fourth is whether MVTec-AD and VisA localization stays within a point of VPDM with one untuned mask family. If the first check fails because real inpainters are badly miscalibrated, the finding still has value: it measures, for the first time, how far diffusion inpainting posteriors are from the exchangeability that both anomaly scoring and on-manifold explanation implicitly assume.