Verify-and-Refine: Proposal-Conditioned Hybrid Losses for Anomaly Segmentation Without Real Defects

Factories and hospitals want a model that outlines where a defect is, not only whether a part is defective. They almost never have labeled defects to train it on. The strongest current methods sidestep the lack of labels in two ways. Some compare every image patch against a memory of normal patches and flag the ones that look unfamiliar. Others paste fake defects onto normal images and train a segmenter to find them. Both work on the whole image at once. The result is that tiny defects are swamped by background, and normal but busy regions, such as object edges and printed traces, raise false alarms. We propose a two-stage alternative borrowed from object detection. A training-free normality model first proposes a handful of suspicious regions. A small learned network then zooms into each region and makes two nested yes/no decisions: does this crop contain a defect, and if so, which pixels.

The part that is new is the change of training regime this cascade makes possible, not the cascade itself. Losses that reward overlap between the predicted and true defect region, such as Dice (twice the overlap divided by the summed areas), are known to help segment small targets. They behave badly in anomaly training, though. On full images they are either undefined, because most images contain no defect, or they swing wildly, because a 5-pixel defect in a 1024×1024 image moves with every single pixel. Cropping to proposals makes defects occupy a large share of the input. Splitting the decision into "is there a defect here" and "outline it" also means the overlap loss is only ever applied where a defect exists. We claim that this combination produces better defect outlines than any of its parts alone, especially for small defects, and that it removes the edge-induced false alarms of memory-based detectors.

Core Claim

A cascade of a nominal-only proposal stage and a crop-level verify-then-segment stage outperforms full-image anomaly segmenters on localization precision. The cascade is trained only on synthetic defects placed inside real proposals, and it uses a distribution loss for the crop decision and a hybrid distribution-plus-region loss for the pixel decision. The gain should come from three identifiable sources: region losses become well-conditioned on crops, the verifier learns to reject the specific false alarms the proposal stage makes, and the hierarchy keeps Dice away from empty masks. The prediction is falsifiable. If adding Dice to a full-image segmenter recovers most of the gain, or if random crops work as well as proposal crops, the claim fails.

How This Relates to Prior Work

Two methods motivate the design by showing what full-image approaches leave on the table. PatchCore (Towards Total Recall in Industrial Anomaly Detection) scores each patch by its nearest-neighbor distance to a coreset of normal ImageNet-pretrained features. It reaches 99.6% image-level and 98.1% pixel-level AUROC on MVTec AD with no training on defects. It has no mechanism for rejecting patches that are unusual but normal. DRÆM (DRAEM - A Discriminatively Trained Reconstruction Embedding for Surface Anomaly Detection; also DRÆM) pastes Perlin-noise-shaped texture patches onto normal images. It trains a full-image U-Net with focal loss alone and reports 68.4 localization average precision (AP), against 55.0 for PaDiM. We reuse PatchCore unchanged as our proposal stage and DRÆM's defect synthesizer as our supervision source. Unlike DRÆM, we segment crops rather than images and add a region loss. Unlike PatchCore, we add a learned verifier.

Several methods already call themselves "two-stage", but none decomposes the problem spatially. UTAD (Unsupervised Two-Stage Anomaly Detection) chains two reconstruction networks to avoid "healing" defects and reports 0.53 mean IoU. ToCoAD (ToCoAD: Two-Stage Contrastive Learning for Industrial Anomaly Detection) runs two stages of feature adaptation before a PatchCore memory bank and reports 98.21% pixel AUROC on MVTec AD and 98.43% on VisA. In both, the stages are sequential training or reconstruction steps. Neither proposes regions and then refines them. ToCoAD matters as an orthogonal improvement: its adapted features could replace our proposal stage's backbone. SPot-the-Difference (SPot-the-Difference Self-supervised Pre-training for Anomaly Detection and Segmentation) acts as a constraint. It shows that invariance-seeking pretraining suppresses local defect cues, so the refinement network's backbone should not be a globally contrastive encoder.

The cascade idea itself comes from supervised work. R-CNN (Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation) separated class-agnostic proposals from per-region classification. The cascaded liver-lesion network (Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields) raised liver Dice on 3DIRCADb from 72.9% with a single U-Net to 93.1% by cascading. Both assume dense labels for every stage. Our stage 1 needs no labels, and our stage 2 sees only synthetic ones.

The hierarchical-decomposition literature supplies the verify-then-segment split and its main risk. HD-CNN (HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition) routes inputs through a coarse classifier to fine experts and relies on soft, overlapping routing to limit irrecoverable early errors. We take the same concern seriously by measuring how much defect area the proposals miss. We deliberately do not attempt a taxonomy of defect types. Without labeled defects that task is poorly posed, and the file offers no benchmark protocol for it.

The hybrid-loss work supplies the loss recipe. Focal plus Dice plus IoU losses on polyps (Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentation) and binary cross-entropy (BCE) plus Dice on white matter hyperintensities (Deeply supervised network for white matter hyperintensities segmentation with transfer learning) help on small targets. Both are fully supervised, and every training image contains foreground. The anomaly setting breaks that assumption, which is the reason for the hierarchy.

Research Program

Stage 1 is PatchCore run as published on nominal training images. Its anomaly map is thresholded at a deliberately low, recall-oriented level. We then take the top $k$ connected regions, each dilated by a fixed context margin so that thin cracks are not cut off at the box edge, and resize each to a fixed crop size.

Stage 2 training data is built by running stage 1 on two kinds of images. Nominal training images give proposals that are PatchCore's own false alarms, which become hard negatives. Images with DRÆM-style synthetic defects give proposals that overlap the pasted defect, which become positives with a known mask. The refinement network is a small U-Net. It sees the crop, the PatchCore distance map for the crop, and a nearest-normal-patch reconstruction, echoing DRÆM's pairing of input and reconstruction. It has two heads. A crop head outputs the probability $q$ that the crop contains a defect. A mask head outputs per-pixel probabilities $p_i$.

For a crop with defect indicator $c$ and mask $y$, the training loss is

$$L = \mathrm{FL}(q, c) + c\left[\lambda\, \overline{\mathrm{FL}}(p, y) + (1-\lambda)\left(1 - \frac{2\sum_i p_i y_i}{\sum_i p_i + \sum_i y_i}\right)\right].$$

Here $\mathrm{FL}$ is focal loss, $\overline{\mathrm{FL}}$ is pixel-averaged focal loss, and $\lambda$ trades distribution against region terms. On negative crops the mask head is still driven toward zero through a light pixel focal term. The Dice term is never computed on an empty mask. This gating is the "hierarchical binary" element: a crop-level yes/no decision, then a pixel-level one conditioned on it.

At test time, a pixel's score is $q \cdot p_i$ inside proposals. Outside proposals it is PatchCore's score clipped below the proposal threshold, so that full-map metrics remain defined and missed proposals are penalized honestly.

The analysis has three parts. We measure the stage-1 recall ceiling, meaning the fraction of ground-truth defect pixels covered by any proposal, as a function of $k$ and threshold. We track gradient-norm variance of the Dice term on full images versus crops, to test the conditioning argument directly. We stratify all localization metrics by defect area, where the claimed gains should concentrate.

Baselines

The naive reference is PatchCore alone, reproduced with the same backbone, coreset ratio and resolution as our stage 1. Its reported numbers are 99.6% image AUROC and 98.1% pixel AUROC on MVTec AD. No reported result was found on Lacuna for its pixel AP, which we therefore measure ourselves. Since our system is PatchCore plus a refiner, any gain over this reproduction is attributable to stage 2.

The strongest competitor on the localization question we target is DRÆM. It is trained on the same synthetic-defect distribution and reports 68.4 localization AP on MVTec AD, the highest pixel-AP figure in the file. We rerun it with its public protocol and the same synthesizer and training images. Our stage 2 gets no more optimizer steps than DRÆM's discriminative network, and hyperparameter search budgets are equal. Beating it means exceeding its reproduced pixel AP on MVTec AD, and exceeding it by more than seed-to-seed variation on the smallest-defect stratum. On VisA, no reported result was found on Lacuna for DRÆM pixel AP, so both are measured. ToCoAD's 98.21% (MVTec AD) and 98.43% (VisA) pixel AUROC serve as a reference ceiling for pixel AUROC. It is also an alternative stage-1 backbone: if ToCoAD features plus our refiner beats ToCoAD alone, the refiner is complementary to feature adaptation.

Three ablations isolate the ingredients:

  • Full-image hybrid loss: DRÆM trained with our exact focal-plus-Dice loss at full resolution. This asks whether the loss alone explains the gain.
  • Random crops: our stage 2 trained on crops centered at random locations and at synthetic defects, instead of at PatchCore proposals. This asks whether proposal-conditioned hard negatives matter or only zooming does.
  • No hierarchy: our cascade with the crop head removed and Dice applied to all crops, with an epsilon smoothing for empty masks. This tests the gating.

Why not just…?

Why not just add Dice to DRÆM's focal loss? This is the most serious threat, because it costs one line of code. Our argument is that on full images most synthetic masks are tiny and many training images are empty, so the Dice gradient is either undefined or dominated by single-pixel flips. The file notes this instability for defects of 1–5 pixels, but we found no paper in it that tests it on anomaly training. The full-image hybrid-loss ablation settles the question. If it closes most of the gap to our cascade, we will report the paper as a loss result and drop the cascade claim.

Why not just post-process PatchCore maps, or rerun PatchCore at higher resolution inside its own proposals? Test-time zoom without learning is a legitimate cheap alternative and we include it as an extra baseline. It can sharpen boundaries. It cannot, however, learn that a high nearest-neighbor distance on a textured normal edge is benign, because the memory bank has no notion of "unusual but acceptable". The hard-negative rejection rate on nominal test images, meaning the fraction of false-positive proposals the crop head suppresses, measures exactly this. It is the number that zoom-only PatchCore should not match.

Why not just train a crop-level segmenter on random or defect-centered crops, skipping the proposal stage during training? If this works as well as proposal-conditioned training, then the proposals only matter at test time, and our "learn PatchCore's mistakes" story is wrong. The random-crop ablation tests it directly, with the same test-time proposals for both variants.

Why not just adapt the features, as ToCoAD does? Feature adaptation and region refinement address different failures, so we treat ToCoAD as a stronger stage 1 rather than a rival. If the refiner adds nothing on top of ToCoAD features, the refiner's value is limited to weak backbones, and we will say so.

Proposed Experiments

Main experiments use MVTec AD (15 categories, nominal-only training, 1,888 annotated defect regions) and VisA (12 categories, including multi-instance and PCB layouts where edge false alarms should be common). BTAD serves as a transfer check. Metrics are pixel AP as the headline, per-region overlap, pixel AUROC for comparability, image AUROC to confirm detection is not harmed, and latency per image against PatchCore's reported 0.17–0.22s. All metrics are also stratified into area terciles of ground-truth defect regions. Each configuration runs with three seeds.

Four checks are decisive.

  1. Proposal recall. Stage-1 recall of defect pixels must exceed the final pixel recall we claim. If the proposals miss too much, the cascade is capped and we will show where.
  2. Beating DRÆM. The cascade must beat reproduced DRÆM pixel AP, with the margin largest in the smallest-area tercile. A uniform gain would argue against the conditioning explanation.
  3. Losing each ingredient. Each of the full-image hybrid loss, random crops and no-hierarchy ablations must lose to the full system. Whichever loses least tells us which ingredient carries the result.
  4. Few-shot adaptation. On VisA's 5-shot and 10-shot splits, adding the few real defects as positive crops should help the refiner more than it helps DRÆM. This probes the synthetic-to-real gap: a crop-level model sees real defects at the same scale as it will at test time.