Shift-Aware Whitening for Defect Detection That Keeps Its Threshold

A common way to automate visual quality inspection works like this. A pretrained network turns image patches of good parts into feature vectors, a Gaussian (a bell-curve model) is fitted to those vectors, and a new patch is flagged when its Mahalanobis distance from the fitted Gaussian is large. Mahalanobis distance is ordinary distance measured after whitening, that is, after rescaling the features so every direction has unit variance and no correlation with the others. This works very well on clean benchmarks, with near-perfect scores on several factory categories. We argue that the same whitening becomes the main weakness once lighting, optics or camera pose drift. Whitening divides each direction by its standard deviation on the training images. Directions that barely moved during training therefore get the largest amplification. Natural drift tends to push good parts along exactly such quiet directions, so harmless changes look like enormous anomalies, and the alarm threshold set on day one stops meaning anything.

The proposal is to split the inspection decision into two binary questions that are answered separately and calibrated separately. The first question is whether this patch has moved in a direction the environment is known to cause. The second is whether it has moved in any other direction. The environmental directions are found without a single defect label. We use only a handful of unlabeled images of good parts taken under the new conditions, and solve one generalized eigenvalue problem. The first question becomes a "the line has drifted" signal for operators. The second becomes a defect score whose rejection threshold, set once from good parts, should keep its false-alarm rate under drift. Automation was supposed to provide this property in place of human inspectors whose acceptance criteria wander over a shift. In practice, current detectors quietly lose it.

Core Claim

Gula and Bertoldo's Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection showed that on MVTec AD the useful directions in whitened feature space are not the high-variance ones, and that fewer than ten well-chosen components can reach 100% AUROC on categories such as bottle and wood. Their selection needs labeled defects and assumes a static environment. We make a narrower and testable claim. Under natural shift, the directions that should be removed are the ones along which good parts move between conditions, measured relative to how much good parts vary within a condition. These directions can be estimated from a few unlabeled good-part images in the target condition. Removing or down-weighting them restores ranking quality (AUROC). It also restores something AUROC cannot see: the false-alarm rate at a threshold fixed in advance.

Let $x$ be a patch feature, $\mu$ and $\Sigma$ the mean and covariance of good-part features in the source condition, and $S_b$ the covariance of the displacement between target-condition good-part features and $\mu$. The nuisance directions are the top-$r$ solutions of

$$S_b v = \lambda \Sigma v,$$

where a large generalized eigenvalue $\lambda$ means the environment moves good parts far along $v$ compared with their normal spread. Writing $W=\Sigma^{-1/2}$ for the whitening map and $P_\perp$ for the projection that removes those $r$ directions after whitening, the two scores are

$$s_{\text{env}}(x)=\lVert (I-P_\perp)W(x-\mu)\rVert^2, \qquad s_{\text{def}}(x)=\lVert P_\perp W(x-\mu)\rVert^2.$$

If good-part features were Gaussian, $s_{\text{def}}$ would follow a chi-squared distribution with $D-r$ degrees of freedom, where $D$ is the feature dimension. That gives a principled threshold from good parts alone. In practice we set the threshold from an empirical quantile on source good parts and test whether it keeps its meaning after drift. A soft variant shrinks nuisance directions by a factor $1/(1+\lambda)$ instead of removing them. This is algebraically equivalent to whitening with the pooled covariance $\Sigma+S_b$, and it sits at one end of the comparison below.

Research Program

Diagnostic phase. We first test the mechanism before building on it. Using a frozen ImageNet-pretrained backbone with patch-level Gaussians, as in PaDiM-style detectors that Real-IAD reports at 84.6% image-level AUROC, we take target-condition good parts and decompose their whitened displacement onto the source eigenbasis. The hypothesis predicts that most of the false-alarm mass lies in source directions with small eigenvalues and lies in a low-dimensional subspace. If instead the displacement is spread evenly over many directions, a low-rank nuisance model is the wrong tool, and the project stops there.

Estimator under few samples. The research file notes that covariance estimation is unstable when samples are few relative to dimension. The design relies on not re-estimating $\Sigma$ in the target condition. We only estimate the low-rank $S_b$. Each image contributes thousands of patch vectors, and the subspace is shared across spatial positions. We will measure how subspace quality, defined as the principal angles to a subspace estimated from all target good parts, degrades as the number of target images $K$ falls from 50 to 1, and how $r$ should be chosen. Our proposal is a stopping rule: keep adding directions until held-out target good parts pass the source threshold at the nominal false-alarm rate. This rule uses no defects.

Leakage and hijacking. Robust Distribution Alignment for Industrial Anomaly Detection under Distribution Shift found that aligning domains can map up to 46.4% of defect patches onto the normal distribution, which it calls anomaly hijacking. Our method has an analogous risk: if a defect type happens to lie along a nuisance direction, removing that direction hides it. Because $s_{\text{env}}$ is kept rather than discarded, we can measure the risk directly as the fraction of each defect's whitened energy that falls in the removed subspace. The decision rule is designed to fail safe. A part is accepted only if $s_{\text{def}}$ is below threshold. A high $s_{\text{env}}$ never routes a part away from the defect check; it only raises a drift flag. We also consider an extreme-$s_{\text{env}}$ escalation rule for parts whose nuisance score is far outside what the target good parts produced.

Relation to hierarchical classification. Cascades such as the organ-then-lesion pipeline in Automatic Liver and Lesion Segmentation in CT, the stage-wise feature selection of hierarchical SVMs for eating-condition speech, and the multi-stage glass pipeline of Smart-Inspect motivate the design. They show that the coarse and fine decisions in an inspection problem need different features. Here the two binary nodes use complementary subspaces of the same whitened space.

ReCAPA constrains the design. It documents how early errors snowball in sequential pipelines, which is why our first node never makes a hard routing decision.

Two-Stage Block-Based Whitened PCA is the closest precedent for two-stage whitening. It recomputes a local whitened PCA for each query in face recognition. We reuse the idea of separate whitening per stage, but our subspaces are fixed per condition rather than per query, so inference costs one extra matrix product.

HaST-CW aligns whitened axes with a hand-built concept tree. We need no tree and no concept labels, and the split we learn is environment versus everything else rather than a semantic taxonomy.

An apparent contradiction. Isotropic Representation Can Improve Dense Retrieval found that whitening improves out-of-distribution retrieval. This is consistent with our account rather than opposed to it. In retrieval, whitening helps by suppressing a few dominant outlier dimensions. In Mahalanobis anomaly detection, the same inverse scaling enlarges the quiet dimensions where drift lands. In both settings the result depends on which directions get amplified, and that is the quantity our generalized eigenproblem controls.

Baselines

Do-nothing reference. The do-nothing reference is the source-fitted patch Gaussian with full-spectrum Mahalanobis scoring, the starting point of Gula and Bertoldo. It shares our backbone, feature layers and source data exactly. It defines how much drift hurts. Beating it is necessary but not interesting.

Pooled-covariance whitening. This refits the Gaussian on source good parts plus the same $K$ target good-part images with Ledoit-Wolf-style shrinkage. It is the cheapest serious competitor. It is also mathematically the soft limit of our method, so it is run with identical $K$ and identical shrinkage tuning, using source-only validation.

Training-based shift methods. The strongest existing methods are trained: FiCo (Filter or Compensate), GNL (Anomaly Detection under Distribution Shift, with the ADShift code from mala-lab), and RoDA. The reported figures to compare against are the following:

  • RoDA reports a mean AUROC of 91.34% on MVTec under level-5 corruptions and 71.00% on corrupted Real-IAD.
  • FiCo reports a 97.6% average AUROC on MVTec AD and outperforms PNI by 19.4% under corruption.
  • GNL reports roughly 0.90 AUROC at the highest severities.

On the natural cross-condition splits below, no reported result was found on Lacuna for any of these methods. We will therefore rerun GNL from its released code. Where FiCo and RoDA cannot be faithfully reproduced, we quote their corrupted-MVTec numbers under their own protocol and note that they serve only as reference points. RoDA's adaptation also uses unlabeled target data, so it receives the same $K$ target images we use.

On the standard corrupted MVTec protocol, "beating" means matching RoDA's 91.34% within noise while adding no training. On natural splits, it means a higher AUROC and a target false-alarm rate closer to nominal than every competitor.

Oracle and ablations. The labeled greedy selection of Gula and Bertoldo serves as an oracle ceiling, since it uses defects we do not. The key ablation removes the same number $r$ of directions chosen in three other ways: at random, by highest source variance (classic PCA truncation), and by lowest source variance. If any of these matches the shift-defined directions, the generalized eigenproblem is not the active ingredient.

Why not just…?

Why not just refit or pool the Gaussian on target good parts? This is the real threat, and it is the pooled-covariance baseline above. Our method and pooling differ only in whether nuisance directions are removed or shrunk, and in whether the nuisance score is kept as a separate signal. If pooling matches our AUROC and calibration at every $K$, the contribution narrows to the diagnosis plus the drift flag, and we will report it that way. We expect a gap at small $K$. Pooling with one to five images barely moves a covariance estimated from hundreds of images, so the shrinkage factor stays near one, whereas the generalized eigenproblem isolates the displacement regardless of how little target data enters $\Sigma$. The $K$-sweep settles this question.

Why not just augment training with synthetic corruptions? Measuring Robustness to Natural Distribution Shifts found that across 204 models, synthetic-robustness training does not lift natural-shift accuracy above the clean-accuracy trend. 3D Common Corruptions attributes this to 2D filters ignoring scene geometry. In anomaly detection specifically, GNL already includes AugMix and is one of our baselines. Our method uses no augmentation, so it cannot overfit to a corruption generator. Equally, results on corrupted MVTec alone will not be taken as evidence of natural robustness.

Why not just train an invariant detector like FiCo or GNL? These methods do address shift. They require retraining per product, and GNL reports roughly 2.8 times the training cost. None of them gives a threshold that stays valid without defect samples, which is the obstacle MVTec AD identifies as unsolved. Our method is closed-form and can be updated whenever the line changes. That matters for embedded deployment, where LightDefectNet shows millisecond budgets are the norm. If GNL wins on AUROC but not on fixed-threshold false alarms, both findings get reported.

Why not just use standard domain adaptation? WILDS found that CORAL and IRM fail to beat plain training on 8 of 10 real-world shifts. RoDA's hijacking result shows that naive alignment hides defects. Our leakage measurement tests whether our projection does the same.

Proposed Experiments and Decisive Checks

Natural shift comes first, because the research file warns that synthetic gains do not transfer.

Real-IAD. We use its five camera views to build cross-condition splits: fit on oblique views, adapt with $K$ unlabeled good images from a held-out oblique view, and test on that view. The fully unsupervised setting, where 10% of training images are defective, tests whether contaminated target images corrupt the subspace estimate.

MVTec 3D (RGB only). This adds acquisition-noise shift.

LGPSDD. Its brightness diversity is split by image intensity into condition bins.

Corrupted MVTec AD. This is kept for direct comparability with FiCo, GNL and RoDA.

Metrics. We report image-level and pixel-level AUROC. The headline metric is threshold persistence: set the threshold at 1% false alarms on source good parts, then report the target false-alarm rate and the defect recall at that unchanged threshold. We also report the leakage fraction per defect type and drift-flag AUROC for separating target good parts from source good parts.

The mechanism stands or falls on three checks.

  1. Low-rank diagnosis. It must hold: most target false-alarm mass lies in a subspace with $r$ well below $D$.
  2. Shift-defined directions. These must beat random, top-variance and bottom-variance removal at equal $r$.
  3. Threshold persistence. It must improve over pooled covariance at $K\le 5$ without leakage hiding any defect class above 10% of its energy.

If check 3 fails against pooling, the paper becomes an analysis of why source whitening breaks under drift and how far simple pooling repairs it. That is a smaller but still useful result. Gula and Bertoldo's labeled greedy selection, run in the target condition, bounds how much room is left.

References

  1. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
  2. Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection
  3. Robust Distribution Alignment for Industrial Anomaly Detection under Distribution Shift
  4. Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields.
  5. Automatic classification of eating conditions from speech using acoustic feature selection and a set of hierarchical support vector machine classifiers
  6. Smart-Inspect: Micro Scale Localization and Classification of Smartphone Glass Defects for Industrial Automation
  7. ReCAPA: Hierarchical Predictive Correction to Mitigate Cascading Failures
  8. Two-Stage Block-Based Whitened Principal Component Analysis with Application to Single Sample Face Recognition
  9. Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classification
  10. Isotropic Representation Can Improve Dense Retrieval
  11. Filter or Compensate: Towards Invariant Representation from Distribution Shift for Anomaly Detection
  12. Anomaly Detection under Distribution Shift
  13. Measuring Robustness to Natural Distribution Shifts in Image Classification
  14. 3D Common Corruptions and Data Augmentation
  15. MVTec AD - A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
  16. High-Throughput, High-Performance Deep Learning-Driven Light Guide Plate Surface Visual Quality Inspection Tailored for Real-World Manufacturing Environments
  17. WILDS: A Benchmark of in-the-Wild Distribution Shifts