Many anomaly detectors keep the pretrained network fixed. They store features of small image patches from defect-free training images in a "memory bank" and flag a test patch as anomalous when it is far from everything stored. This works well in controlled factory imagery. It breaks in two linked ways once the scene becomes uncontrolled, with varying viewpoints, backgrounds and lighting. First, the frozen network was never trained to keep that variation apart from real defects, so harmless changes look as distant as cracks. Second, the bank must grow to cover all that harmless variation, and storage and nearest-neighbor search scale with the number of stored patches times the feature dimension. We argue these are one problem. The directions that carry most of the variance among normal patches in an uncontrolled scene are mostly nuisance directions. They inflate the bank, cause false alarms, and are exactly the directions that standard compression such as PCA keeps.
The proposal is to estimate, from normal data and image augmentations alone, a small linear projection of frozen patch features. The projection keeps directions along which genuinely different normal content varies and suppresses directions along which the same content varies under nuisance transformations. The memory bank is then built and searched in this space of a few dozen dimensions. One cheap, closed-form step should cut bank dimension by an order of magnitude, cut the number of stored patches needed for coverage, and raise accuracy under uncontrolled conditions. A byproduct is a pre-deployment diagnostic. If no direction in the frozen features separates content from nuisance, the projection's spectrum says so before any detector is built. That gives a direct test of whether the frozen backbone is representationally adequate for the target domain.
Core Claim
Let $f(x,p) \in \mathbb{R}^D$ be the frozen mid-level feature of patch $p$ in normal image $x$, as in PatchCore. For a nuisance transformation $T$ drawn from a chosen family, let $p'$ be the patch in $T(x)$ that corresponds to $p$. The correspondence is known because $T$ is applied by us. We define a nuisance scatter $S_w$, the covariance of the differences $f(T(x),p') - f(x,p)$, and a content scatter $S_b$, the covariance of normal patch features across the training set. The projection $W \in \mathbb{R}^{D \times d}$ maximizes
$$ J(W) = \frac{\operatorname{tr}(W^\top S_b W)}{\operatorname{tr}(W^\top (S_w + \lambda I) W)}, $$
where $\lambda$ is a small ridge term. As in linear discriminant analysis, the solution is the top $d$ generalized eigenvectors of $(S_b, S_w + \lambda I)$, and it is computed in seconds. The anomaly score is the PatchCore nearest-neighbor distance computed on $W^\top f$.
We make three falsifiable predictions. On controlled data (MVTec AD), the projection matches full-dimensional PatchCore at $d \ll D$. On uncontrolled data (MIAD), it beats both full-dimensional PatchCore and PCA at equal $d$, because PCA's top directions are dominated by nuisance there. The generalized eigenvalue spectrum, meaning how many directions have a content-to-nuisance ratio well above 1, predicts across categories how well any frozen-feature memory method will do.
Positioning Against Prior Work
The motivating evidence comes from two sides.
On the memory side, PatchCore stores frozen ImageNet patch features and reduces only the number of entries by greedy coreset selection, leaving $D$ untouched. CFA learns a patch descriptor with a hypersphere loss and a fixed-capacity bank, but it also stores full-dimensional features and needs multi-epoch training with radius and balance tuning. MIAD shows the collapse we target: detectors fall from near 0.99 AUROC indoors to as low as 0.58 with all outdoor factors, viewpoint and background each costing about 0.2. MIAD also reports that banks must grow to tens of thousands of entries.
On the rigidity side, the PatchCore paper itself and adaptation work such as Path-CLIP and VDPG argue that frozen features cap performance, while fine-tuning risks forgetting and overfitting. Our projection does not try to create features the backbone lacks. It removes nuisance directions the backbone does represent, and it reports when nothing useful remains.
The closest connecting work is MRAD, which retrieves from frozen CLIP feature banks and shows banks can be randomly subsampled heavily. InCTRL caches a few normal images and scores residuals in frozen CLIP space. Both work in the native feature space. Neither separates nuisance from content, and neither reduces $D$.
On compression, On the Intrinsic Dimensionality of Image Representations found intrinsic dimension around 16–19 for 512-dimensional embeddings and reported that its nonlinear DeepMDS mapping retained more accuracy than PCA in face and object recognition. That paper both supplies and constrains our design. It suggests low $d$ is feasible. It also warns that a linear map may lose information. We therefore include a nonlinear variant, but we hypothesize that for patch anomaly scoring the dominant loss comes from which directions are kept, not from linearity. This is a hypothesis we test, not one we assume.
Research Program
We build the estimator for frozen WideResNet-50 patch features, following PatchCore, and for frozen CLIP ViT patch tokens, following MRAD and InCTRL. Nuisance families are photometric (brightness, contrast, color temperature), geometric (rotation, crop, small perspective warps, with patch correspondence from the known warp), and background (pasting the object region onto random textures where a foreground mask can be obtained). We sweep $d$ from 8 to 256 and run greedy coreset selection in the projected space.
We also measure "coverage size", the number of coreset points needed to bring the covering radius of the normal patches below a fixed quantile. This tests whether removing nuisance shrinks $N$ as well as $D$.
The diagnostic is the count of generalized eigenvalues above a threshold, together with the content-to-nuisance ratio of the top $d$. We correlate it with final AUROC across MVTec AD, MIAD scenarios and factor settings, and medical sets from the InCTRL and MRAD suites. A strong rank correlation would give practitioners an a priori adequacy test for frozen backbones. That test is listed as missing in the rigidity literature.
The main risk is that some defects resemble nuisance: a stain looks like a lighting change, and a misplaced part looks like a viewpoint change. Invariance to such transforms would then erase the defect. We handle this by ablating each nuisance family separately and reporting results by defect type. MIAD distinguishes surface from logical defects, so we can see where invariance helps and where it hurts.
Baselines
The naive reference is full-dimensional PatchCore with its standard coreset ratio, run on our hardware with identical feature extraction. PatchCore reports 99.6% image AUROC and 98.1% pixel AUROC on MVTec AD at 0.17–0.22 s per image. On MVTec, success means staying within about 0.5 points of these numbers at $d \le 64$.
The strongest compression competitor is PatchCore with PCA to the same $d$, fit on the same normal patches, plus a random Gaussian projection to the same $d$. Both are given the same coreset budget. To count as beating them, our projection must give a clear gain at matched $d$ on MIAD and must not lose on MVTec AD.
The strongest adaptation competitor is CFA, which reports 99.5% image AUROC and 98.5% pixel AUROC on MVTec AD, with 0.8% image-AUROC degradation on rotated and cropped RD-MVTec AD. We run CFA with its authors' hyperparameter protocol, and we also run a variant whose descriptor outputs $d$ dimensions. Beating CFA means matching its accuracy on RD-MVTec AD and MIAD at a fraction of its training time and bank size.
On MIAD we also compare against DRAEM and Reverse Distillation, which the MIAD paper found best for surface and logical defects respectively. Per-method numbers for them are not given in this file beyond the overall drop to as low as 0.58 AUROC, so no reported result was found on Lacuna for the per-scenario figures, and we rerun both.
The ablation that isolates our key ingredient whitens features by $S_w$ alone at full $D$, which gives invariance without compression. A second ablation keeps the top content-variance directions without nuisance whitening, which is PCA. Together they separate the contribution of invariance from that of dimension.
Why not just…?
Why not just use PCA, random projection or product quantization to shrink the bank? On MVTec AD these may well work as well as our projection. That outcome is consistent with our claim, and we will report it plainly. The disagreement is on uncontrolled data. There the highest-variance directions are viewpoint, background and lighting, so PCA keeps exactly what we want to discard. Quantization compresses whatever directions exist and therefore keeps the nuisance distances too. The matched-$d$ PCA and random-projection comparison on MIAD settles this. If PCA matches us there, the core claim is false.
Why not just use CFA or another learned patch descriptor? CFA adapts features to the target but does it by pulling normal features into compact clusters. It does not explicitly target nuisance, it stores full-dimensional entries, and it needs iterative training and tuning. If CFA at reduced output dimension beats our closed-form projection on MIAD, the nuisance-scatter idea adds little beyond generic adaptation. Our CFA-with-$d$-outputs baseline tests exactly this.
Why not just keep a bigger coreset with an approximate nearest-neighbor index? That attacks latency but not false alarms. MIAD shows accuracy itself collapses as nuisance grows, so more storage of nuisance variation does not restore separation. We plot AUROC against bank size for full-dimensional PatchCore on MIAD. If AUROC keeps climbing toward our level as the bank grows, this alternative is a real threat, and our contribution reduces to an efficiency gain.
Why not just fine-tune the backbone? Fine-tuning on normal-only data has no anomaly signal and risks collapse. Path-CLIP and VDPG document forgetting and loss of robustness. We do treat fine-tuning as the fallback our diagnostic should recommend: categories with low content-to-nuisance spectra are where we expect frozen methods, including ours, to fail.
Decisive Experiments
The first test runs on MIAD's uncontrolled-factor splits, with AUROC and Recall at 1% false positive rate. We compare projected PatchCore at $d \in \{16, 32, 64\}$ against full-dimensional PatchCore and PCA at the same $d$, and we report bank memory and query latency. The idea succeeds if it shows a consistent AUROC gain over both together with an order-of-magnitude memory cut.
The second test checks that nothing breaks on MVTec AD and RD-MVTec AD. The geometric nuisance family should specifically close the rotation gap.
The third test measures coverage size with and without projection on MIAD. It shows whether nuisance removal reduces $N$, the axis PatchCore and CFA already target, beyond reducing $D$.
The fourth test is the diagnostic: a rank correlation across categories and datasets, including the medical sets in the MRAD and InCTRL suites, between the generalized eigenvalue statistic computed before deployment and the eventual AUROC of frozen-feature detectors.
The fifth test is the per-defect-type ablation of nuisance families. It tells us when invariance erases real defects, the failure mode most likely to limit the method.
References
- Towards Total Recall in Industrial Anomaly Detection
- CFA: Coupled-Hypersphere-Based Feature Adaptation for Target-Oriented Anomaly Localization
- MIAD: A Maintenance Inspection Dataset for Unsupervised Anomaly Detection
- Bridging the Pathology Domain Gap: Efficiently Adapting CLIP for Pathology Image Analysis with Limited Labeled Data
- Adapting to Distribution Shift by Visual Domain Prompt Generation
- MRAD: Zero-Shot Anomaly Detection with Memory-Driven Retrieval
- Toward Generalist Anomaly Detection via In-Context Residual Learning with Few-Shot Sample Prompts
- On the Intrinsic Dimensionality of Image Representations