When only a handful of normal images are available, say two to eight defect-free parts or healthy scans, a detector has very little to go on. One common way to find defects is to reconstruct a "normal-looking" version of the input and flag the regions where the reconstruction disagrees with the actual image. This kind of detector usually fails in a specific way: a powerful reconstruction network copies the defect along with everything else, so the difference map comes out empty. We propose to block this copying with two separate constraints that both make sense with very few samples.
The first constraint is masking. Patches of the image are hidden, and a pretrained masked autoencoder must predict them from the surrounding context. A masked autoencoder is a vision transformer trained to fill in heavily masked images. The second constraint is a sparsifying transform. This is a linear map under which normal content has only a few non-zero coefficients. It is fitted in closed form at test time from the few support images and pulled toward a prior learned on other categories. Each predicted patch is pushed through this transform, all but its largest coefficients are zeroed, and the result is mapped back. Context alone cannot produce a lesion that is hidden by the mask, and the transform cannot represent content that normal images never needed. The anomaly score is the remaining residual.
Core Claim
The claim is narrow and testable. In the few-shot regime, an analysis sparsifying transform fitted to a frozen masked-image-modeling feature space is a better model of "normal" than three alternatives: a nearest-neighbour memory bank, a single Gaussian, or a synthesis dictionary built from raw support features. We further claim that applying it to masked-context predictions rather than to observed tokens removes most of the residual identical-shortcut failure.
An analysis transform is a matrix $W$ such that $Wx$ is sparse. It differs from a synthesis dictionary $D$ with $x \approx D\alpha$, which requires an iterative search for the code $\alpha$.
We expect the analysis transform to matter for three reasons. It pools structure across all support tokens instead of storing them as individual atoms, so one bad support image should not corrupt it. It models normal data as a union of low-dimensional subspaces, which a single Gaussian cannot do. And it is solved exactly with one singular value decomposition, so adaptation is cheap and involves no meta-learning.
Relation to Prior Work
Two systems already combine masking with reconstruction-based anomaly detection, and they motivate the masking half of our design.
MemMC-MAE adds three things to a masked autoencoder: 75% masking, learnable memory matrices inside self-attention, and a multi-level cross-attention decoder. It reports an AUC of 0.879 on Pneumonia chest X-ray and 0.917 on Covid-X.
MAEDiff embeds a masked autoencoder in a diffusion U-Net. It reports 42.07% DICE on BraTS21 and 9.61% on MSLUB, and observes that global attention without masking hurts.
Both are trained on thousands of images or hundreds of full scans. Both fix their normality model after training. MemMC-MAE's memory is a soft attention over slots, not a sparsity-constrained code. We keep the masking insight but drop training on the target category entirely. Normality is instead fitted at test time from $K \le 8$ images.
The most direct predecessor is Few-Shot Fast-Adaptive Anomaly Detection (Adaptive EBM). It uses support features directly as dictionary atoms in a learned-shrinkage sparse coding layer, meta-trains an energy model with Langevin-synthesized negatives, and needs no parameter updates at test time.
We differ from it on three axes. We use an analysis transform rather than raw-atom synthesis, a frozen masked-image-modeling backbone rather than a meta-learned extractor, and masked-context prediction rather than reconstruction of the observed query. Its dependence on raw support atoms is exactly the noise sensitivity our support-corruption experiment targets.
RegAD provides our evaluation protocol: 2/4/8-shot on MVTec and MPDD, with one model shared across categories. It is also the single-Gaussian baseline whose unimodality assumption we test.
The sparsity machinery comes from transform learning. STROLLR learns a unitary transform with closed-form hard thresholding directly from a single degraded image, with no external training set. That "internal learning" property is what makes a transform fit from four images plausible. Constrained Overcomplete Analysis Operator Learning shows that learned analysis operators beat fixed ones for specific signal classes.
Deep ADMM-Net and SC-VAE show that model-based sparse layers train well from small data. ADMM-Net trained on 100 MR images and transferred to chest MRI. SC-VAE puts LISTA, an unrolled sparse-coding solver, inside an autoencoder. Neither is used for anomaly detection, and we deliberately do not unfold an iterative solver. A unitary transform needs no unfolding, and keeping it closed-form is what keeps adaptation fast.
Finally, BMAD both motivates and constrains the project. It finds that reconstruction methods lag behind embedding methods such as PatchCore because of over-generalization, and that synthetic anomalies like CutPaste fail on medical textures. We therefore avoid synthetic negatives altogether. We also treat the embedding methods, not autoencoders, as the bar to beat.
Method
The backbone is a frozen masked autoencoder, consisting of an encoder $E$ and a decoder $G$ that predicts tokens for masked positions. It is pretrained once on natural images. Medical runs use a second variant pretrained on unlabeled, public scans from domains disjoint from the test cohorts.
On a set of base categories, we fit a prior transform $W_0$ for token features. These categories are the MVTec classes outside the held-out fold, following RegAD's leave-out protocol.
For a novel category with support images, we collect support tokens $X \in \mathbb{R}^{d \times N}$. Here $d$ is the token dimension, reduced to 128 or 256 by a fixed projection learned on base categories, and $N$ is the number of support tokens. We then solve
$$\min_{W^\top W = I,\; Z} \; \|WX - Z\|_F^2 + \lambda \|W - W_0\|_F^2 \quad \text{s.t. each column of } Z \text{ has at most } s \text{ non-zeros}.$$
Here $Z$ holds the sparse codes, $s$ is the per-token sparsity, and $\lambda$ sets how strongly the few-shot transform is pulled toward the base-category prior. The $Z$-step is hard thresholding. The $W$-step is the orthogonal Procrustes solution $W = UV^\top$, where $U \Sigma V^\top$ is the singular value decomposition of $ZX^\top + \lambda W_0$. A handful of alternations converge in well under a second.
At test time we draw $M$ random masks at 75% masking. For each patch $p$ that is hidden under a given mask, we form the decoder's context prediction $\hat{x}_p$ and project it onto the normal model as $\tilde{x}_p = W^\top H_s(W\hat{x}_p)$, where $H_s$ keeps the $s$ largest-magnitude coefficients. The anomaly score of patch $p$ is
$$a_p = \|x_p - \tilde{x}_p\|_2 \; + \; \beta\,\|W x_p - H_s(W x_p)\|_2 .$$
The first term asks whether context plus the normal sparse model can explain the observed token $x_p$. The second term, weighted by $\beta$, asks whether the observed token is itself sparse under $W$. Scores are averaged over masks and upsampled for localization.
For pseudo-healthy visualization, a pixel decoder renders $\tilde{x}_p$. This counterfactual is something embedding-distance methods cannot produce. It is useful for interpretation, but we treat it as secondary, not as the main justification.
Research Program
The work proceeds in three stages.
The first stage builds the transform layer and verifies, on base categories, that normal tokens from a frozen masked encoder are actually compressible under a unitary transform. We measure the energy captured at sparsity $s$ against a PCA basis of equal rank. If a single linear subspace captures as much energy, the union-of-subspaces argument fails, and we report that.
The second stage runs the few-shot protocol across industrial and medical data. It ablates each ingredient: masking versus no masking, analysis transform versus alternatives, and with versus without the prior $W_0$.
The third stage stress-tests the claims about the small-sample regime. We vary $K$, corrupt the support set, and shift the pretraining domain.
Baselines
The naive reference point is the frozen masked autoencoder used as a plain reconstruction detector. It encodes the image with masking, decodes, and scores the residual, with no transform and no support adaptation beyond optional normalization. If our full method does not clearly beat this, sparsity contributes nothing.
The strongest competitor is a few-shot PatchCore-style memory bank built on the same frozen encoder's patch tokens from the $K$ support images, scored by nearest-neighbour distance. BMAD identifies this family as the best performer in medical anomaly detection. It is also, in effect, synthesis coding with one active atom over raw support features. We give it the same backbone, the same feature projection, the same support images and the same hyperparameter search budget as our method, namely a grid over layers and neighbourhood size tuned on base categories only. We found no reported few-shot PatchCore result on Lacuna for MVTec at 2/4/8 shots, so we will run it ourselves.
Two further baselines isolate the choice of normality model on the same backbone. The first is a RegAD-style multivariate Gaussian with Mahalanobis scoring. The second is an Adaptive-EBM-style synthesis layer that uses support tokens as dictionary atoms with soft-threshold coding. For RegAD as published, the file reports only relative gains of 5.1%, 6.9% and 8.0% average AUC over TDG+ and DiffNet+ at 2/4/8 shots on MVTec. For Adaptive EBM, it reports no exact AUC: no reported result found on Lacuna for Adaptive EBM numeric AUC. We will therefore rerun RegAD's released protocol rather than quote absolute numbers.
As full-data upper references, we quote DRÆM's image-level AUROC of 98.0 and pixel AP of 68.4 on MVTec from DRÆM. On chest X-ray we quote MemMC-MAE's 0.879 AUC on Pneumonia, which was trained on full data.
A key ablation removes masking but keeps the transform, scoring $\|x_p - W^\top H_s(Wx_p)\|$ on observed tokens. Another removes the prior by setting $\lambda = 0$.
"Beating" the strongest competitor means two things. First, higher mean image AUROC and pixel AUROC/PRO than the memory bank at $K = 2$ and $K = 4$, with category-level paired tests over five support-set seeds. Second, a smaller performance drop under support corruption. Matching it at $K = 8$ alone would not count.
Why not just…?
Why not just use a nearest-neighbour memory bank on frozen masked-autoencoder features? This is the real threat, and the proposal could lose to it. BMAD shows embedding methods beating reconstruction. With good frozen features, nearest-neighbour scoring may already be near the ceiling on MVTec. Our bet is specific to tiny $K$. With 2–4 images, a memory bank has no way to generalize to normal variants missing from the support set, such as a rotated screw or a slightly different gland shape. A transform fitted under a base-category prior can still represent such variants as sparse combinations. The head-to-head comparison at $K = 2, 4, 8$ on identical features settles this. If the memory bank matches us at every $K$ and under support corruption, we will report the method as offering only pseudo-healthy visualization, not better detection.
Why not just reconstruct masked patches with a plain masked autoencoder? MemMC-MAE needed memory matrices in addition to 75% masking because plain masked autoencoders still over-generalize. Its authors added that machinery precisely to stop anomalies leaking into reconstructions. A frozen, generically pretrained decoder has no notion of the target category's normality at all. The naive baseline and the no-transform ablation measure this directly.
Why not just fit a Gaussian or a PCA subspace to the support tokens? This is cheaper still, and it is what RegAD does. It assumes normal tokens form one elliptical cluster. Industrial objects and anatomy are mixtures of distinct part types, such as background, thread and head, or tissue classes. A sparse transform assigns each token its own small set of active coefficients, which amounts to a union of subspaces. The compressibility check in the first stage and the Gaussian baseline together test whether this distinction matters in practice.
Why not just use Adaptive EBM's synthesis dictionary on a better backbone? That swap is our synthesis baseline. The predicted difference is robustness. Raw-atom dictionaries inherit every artifact in the support images, whereas a transform fitted by least squares across all tokens dilutes a single bad image. The corruption experiment below is designed to expose this.
Proposed Experiments
The industrial experiments follow RegAD's protocol on MVTec AD and MPDD at $K = 2, 4, 8$. They report image AUROC, pixel AUROC and pixel AP, averaged over five support draws.
The medical experiments use BMAD's reorganized datasets, including BraTS2021 and RSNA chest X-ray, with the normal training pool subsampled to $K \in \{2, 4, 8, 16\}$ slices or images. They report image AUROC, pixel AUROC and PRO. On brain MRI we additionally report DICE, so that the results can be set against MAEDiff's full-data figures of 42.07% on BraTS21 and 9.61% on MSLUB, with the explicit caveat that our method sees far less data.
Four checks decide whether the claim holds.
The first is a shortcut test. On MVTec test defects, we measure the fraction of anomalous-patch energy that survives in $\tilde{x}_p$ in four conditions: masking only, transform only, both, and neither. The claim requires that the two constraints together suppress this energy substantially more than either alone.
The second is a support-corruption test. We replace one of $K = 4$ support images with a defective image, or with an image from another category. We then compare the drop in AUROC for the transform, the synthesis dictionary and the memory bank. The claim predicts the transform degrades least.
The third is a $K$-scaling curve from 1 to 16 shots. It shows where the prior $W_0$ helps, which we expect at small $K$, and where it stops mattering.
The fourth is a pretraining-domain test. It compares natural-image pretraining with masked pretraining on unlabeled medical data on the BMAD tasks, addressing BMAD's observation that natural-image features help but may miss subtle, low-contrast findings.
Adaptation and inference times will be reported against RegAD's 4.47-second adaptation time. If the method wins on corruption robustness and at $K \le 4$ but not at $K = 8$, that is still a useful, honestly bounded result about when sparse models of normality are worth having.
References
- Unsupervised Anomaly Detection in Medical Images with a Memory-Augmented Multi-level Cross-Attentional Masked Autoencoder
- MAEDiff: Masked Autoencoder-enhanced Diffusion Models for Unsupervised Anomaly Detection in Brain Images
- Few-Shot Fast-Adaptive Anomaly Detection
- Registration based Few-Shot Anomaly Detection
- The Power of Complementary Regularizers: Image Recovery via Transform Learning and Low-Rank Modeling.
- Constrained Overcomplete Analysis Operator Learning for Cosparse Signal Modelling
- Deep ADMM-Net for Compressive Sensing MRI
- SC-VAE: Sparse coding-based variational autoencoder with learned ISTA
- BMAD: Benchmarks for Medical Anomaly Detection
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection