Reconstructing From the Deployment Site, Not the Training Site

Anomaly detectors that work by reconstruction learn what "normal" looks like from defect-free examples. At test time they redraw each input as its closest normal version and flag whatever they could not redraw. The weakness this note targets is that "normal" is learned from one place and one period. A model trained on one tunnel, one stretch of ocean in one season, or one set of power-line photographs absorbs the local background as part of normality: the concrete texture, the lighting, the sea state, the amount of weathering. When it is moved to another site or a later date, it cannot redraw the unfamiliar but harmless background. Those residuals then appear as false alarms. Evidence for this failure is scattered across the field. Diffusion reconstruction collapses to 71.89% pixel AUROC on a real road-tunnel wall because rock and concrete textures cannot be "healed" (Marc et al.). Power-line detectors confuse ordinary weathering with structural defects (CableInspect-AD). Detectors trained on one domain lose 14–34% when moved to another (ADNet).

We propose a different contract for the reconstruction model. Instead of storing normality in its weights, the model receives a handful of defect-free images collected at the deployment site and time, and must reconstruct the test image using only those as its reference for local appearance. Training is done in episodes drawn from many different sites and periods, so the weights learn how to borrow appearance from a reference set rather than what any particular site looks like. The test is narrow and falsifiable. When only a few local normal images are available, this context-conditioned reconstruction should beat both a frozen detector and the obvious fix of dropping those same few images into a nearest-neighbor memory bank. It should beat them on held-out sites and on later time periods.

Core Claim

Let $s$ index a scope, meaning a site and a time window. Let $C_s = \{c_1,\dots,c_K\}$ be $K$ defect-free images from that scope, with $K$ small (4 to 32). For a test image $x$ from scope $s$, a generator $G_\theta$ produces $\hat{x} = G_\theta(x; C_s)$. The anomaly map is a feature-space distance $a(x) = d(\phi(x), \phi(\hat{x}))$, where $\phi$ is a frozen pretrained encoder. We use feature space rather than raw pixels because pixel subtraction is known to fire on misalignment and sensor grain (THFR, ReDi).

The claim has two parts. First, when the scope of the context set matches the test image, false positives on harmless local appearance drop substantially relative to using a context set from the training scope. Second, at small $K$, the learned generator uses the context better than a non-parametric patch memory bank built from the same $K$ images. The second part is what separates this proposal from simply refreshing a memory bank.

Relation to Prior Work

Three lines of work supply the method or come closest to it. ReDi retrieves a similar normal "prompt" image to steer reconstruction and reports 98.8% detection AUROC on MVTec AD. THFR conditions feature restoration on nearest neighbors from a normal template bank, reaching 74.1 sPRO on MVTec LOCO AD. In both, the reference bank is built at training time from the training scope. Reference conditioning there handles pose and layout, not a change of site. We reuse their idea that reconstruction can be guided by retrieved normals, but we change where the references come from and train explicitly for references from a scope the model has never seen.

CORE is the closest in spirit. It reconstructs tabular rows using retrieved normal samples as in-context examples and generalizes across 34 datasets without retraining (average AUROC 0.8488). CORE works on tables, has no spatial anomaly maps, and treats a dataset as the unit of shift. We carry the in-context idea to images, define the unit of shift as a physical site and time window, and measure localization.

DRÆM supplies the training signal we reuse. It pastes synthetic Perlin-noise anomalies onto normal images and trains a reconstruct-then-segment network, reaching 98.0 image AUROC on MVTec AD.

Two papers constrain the design. FiCo removes nuisance style through normalization and filtering and achieves 97.6% AUROC on MVTec AD with synthetic blur and noise. It was never tested on physical site or season shift. It also tries to discard local appearance, whereas we try to explain it from local references. That difference matters when the nuisance is spatially structured, such as bolts, cables, or snow, rather than a global style. ADNet routes features through experts chosen by image semantics and shows that one detector covering many categories degrades: 90.6% falls to 78.5% I-AUROC, recovered to 83.2% by Dinomalym. This warns us that a single generator trained across many scopes may blur them together. The context set is our answer to that risk, and it is one we must test.

Finally, the diffusion results constrain the choice of generator. On tunnels (Marc et al.) and on SAR speckle (Hanssen et al., where an autoencoder at F1 0.639 beat a diffusion model at 0.610), diffusion did worse than simpler models. We therefore start with a U-Net-style generator that cross-attends to context features. We treat diffusion as an optional variant, not the default.

Research Program

The model is a reconstruction network that cross-attends from the test image to encoder features of the $K$ context images, followed by a small DRÆM-style head that segments anomalies. Each training episode does three things. It samples a scope, draws $K$ context images and a separate query image from that scope, and pastes a synthetic anomaly into the query. The loss is

$$\mathcal{L} = \lVert \phi(G_\theta(\tilde{x}; C_s)) - \phi(x) \rVert_2^2 + \lambda\,\mathrm{Focal}(M_\theta(\tilde{x}, \hat{x}), m),$$

where $\tilde{x}$ is the corrupted query, $x$ the clean query, $m$ the pasted-anomaly mask, $M_\theta$ the segmentation head, $\mathrm{Focal}$ a focal segmentation loss, and $\lambda$ a weight.

Two design rules target the "identity shortcut," where a high-capacity network simply copies its input, defect included. First, the query never appears in its own context. Second, the bottleneck on the query path is narrow, so fine appearance has to come from the context. Training scopes come from ADNet's many source datasets and categories, which provides the diversity of "places."

The second deliverable is an evaluation protocol that holds out scope. The datasets in the file are rarely organized by site and time, so this is necessary rather than cosmetic. Four scope-held-out settings are planned.

ADNet is split leave-one-domain-out: train on four domains, test on the fifth, with $K$ normal images from the held-out category as context.

The single-site Tunnel Wall Dataset becomes a pure target scope. Nothing tunnel-specific is used in training, and $K$ of its 107 sane images serve as context.

CableInspect-AD is split by cable type, and by acquisition condition where the annotations allow it. Its weathering cases are scored separately, as a direct probe of the gap between wear and defect.

The Sentinel-1A oil-spill scenes are split by acquisition date, earlier scenes for training and later scenes for testing. We will also split by region if the scene metadata supports it. If neither is recoverable, we fall back to splits that keep whole scenes apart and report that the temporal test could not be run.

A time-specific risk is that a context set drawn from recent observations absorbs a slowly growing defect into "normal." We test this by building context sets that deliberately include early-stage anomalous images, and by measuring how much detection degrades as contamination rises.

Baselines

The naive reference point is a detector trained on the source scopes and applied unchanged to the target, with no target images at all. We instantiate it with DRÆM and with Reverse Distillation. All methods use the same backbone and the same training data, and receive matched compute and an equal hyperparameter-search budget tuned on source scopes only. The proposal should beat this reference by a wide margin. If it does not, scope shift is not the bottleneck on these splits.

The strongest within-scope baselines come from the tunnel benchmark. MSFlow reports 92.22% pixel AUROC and 88.52% AUPRO there, and Reverse Distillation reports 91.57% (Marc et al.). Both were trained on all 107 sane images from that same tunnel. That is an upper-bound regime with full access to the scope, not a competitor at small $K$. We report how close we come using $K \le 32$ local images and zero tunnel training. Reaching within a few points would count as success.

The strongest competitor on the actual question is a nearest-neighbor patch memory bank refreshed with the same $K$ target images, implemented as Enhanced-PatchCore with its nominal-only percentile threshold. It is dangerous because it is also a way to "use local normal images," and it requires no training. CableInspect-AD reports F1 0.75 for it at 100 shots on that dataset. We run it at $K \in \{4, 8, 16, 32, 100\}$ with the same backbone features and the same thresholding rule as our method. Beating it means higher F1 and pixel AUPRO at $K \le 16$ on held-out scopes. At $K = 100$ we expect the two methods to converge.

The robustness baseline is FiCo, trained on source scopes. It represents the alternative strategy of discarding nuisance appearance. Its reported numbers, 97.6% AUROC under synthetic corruption and 62.3% on the PACS Sketch domain, come from synthetic shifts, and we will report how it fares on physical ones.

For the multi-domain setting, Dinomalym is the reference, at 83.2% I-AUROC and 93.1% P-AUROC on ADNet (ADNet). No leave-one-domain-out number for it was found on Lacuna, so we will run it ourselves.

The key ablation keeps our trained model fixed and changes only where the context comes from. In one arm, the $K$ images come from the target scope. In the other, they are $K$ nearest neighbors retrieved from the training bank, which is the ReDi/THFR setting. The gap between the two arms isolates the contribution of scope matching. A second ablation removes context entirely, which recovers a DRÆM-like model, and isolates the contribution of conditioning.

Why not just…?

Why not just refresh a PatchCore memory bank with the local images? This is the real threat. If patch-level nearest-neighbor matching on a few local images already explains local appearance, the generator adds nothing. Our reason to expect a gap at small $K$ is coverage. With four to sixteen images, many harmless local configurations, such as a bolt at a new position or a different crack-free texture patch, have no close neighbor, whereas a generator that has learned to transfer appearance can synthesize them. The $K$-sweep in the Baselines section settles this directly. If the memory bank matches us at every $K$, we will report that outcome. The contribution then reduces to the scope-held-out protocol plus the finding that non-parametric refresh suffices.

Why not just recalibrate the threshold per site? Enhanced-PatchCore already sets thresholds from distances among nominal images (CableInspect-AD). A per-site threshold fixes shifts that move all scores together. It cannot fix false alarms that are spatially localized, such as unfamiliar cables, bolts, and graffiti on tunnel walls (Marc et al.), or lookalike slicks in SAR. We apply per-scope recalibration to every baseline. The pixel-level AUPRO comparison on these localized nuisances shows whether recalibration alone closes the gap.

Why not just fine-tune the detector on the local normal images? Fine-tuning on a few images tends to overfit, and it repeats the scope problem in miniature at each new site. Still, it is a fair competitor. We run DRÆM fine-tuned on the same $K$ target images, with matched optimization steps. The prediction is that it degrades more than our method as $K$ falls, and that it costs per-site training, which our method avoids.

Proposed Experiments and Decisive Checks

The primary metrics are image AUROC, pixel AUROC, and AUPRO on held-out scopes, plus F1 at the threshold derived from nominal images only. For CableInspect-AD we additionally report the false-positive rate on weathered-but-nominal regions. For CableInspect-AD and SAR we also report precision at high-nominal ratios, by subsampling test sets to more than 99% nominal, because field inspection looks like that and the published ~55% anomalous split does not.

The decisive checks are four. The first is the scope-matching ablation: target-scope context must beat training-bank context on held-out scopes, while the two should be roughly equal within scope. If they differ within scope too, the gain is not about scope.

The second is the $K$-sweep against the refreshed memory bank, which decides whether the learned generator is worth having.

The third is the forward-in-time SAR split. It tests whether the approach handles time as well as place. We compare against the autoencoder's reported F1 of 0.639 within its original split (Hanssen et al.) and rerun that autoencoder on our temporal split.

The fourth is the contamination curve, which reports how much anomalous material can enter the local context set before slow defects stop being detected. That number determines whether the method can be deployed on sites that change over time.