Reconstruction-based anomaly detectors work by training a model to rebuild only normal, defect-free examples, so that at test time anything it cannot rebuild stands out as a large error. They have two opposite failure modes. If the model is too expressive, it rebuilds defects as well and the error signal vanishes. If its capacity is restricted, the restriction usually removes whatever varies least in the training data. That is not the same as what matters: lighting, texture and pose variation can survive while thin scratches get discarded or misreconstructed. Current fixes restrict capacity by a number someone chose, such as the rank of a projection or the size of a prototype codebook. Nothing in that number says which directions of variation the model should keep.
This note proposes letting language choose the directions. For each product category we write short descriptions of what normal instances look like and how they normally vary. We embed them with a vision-language model's text encoder, remove the constant offset that separates text embeddings from image embeddings, and use the span of the centered text embeddings as the bottleneck through which image patch features must be reconstructed. The bottleneck's width then comes from how many independent normal factors the descriptions name. Its directions come from meaning rather than pixel variance. Because the offset is removed before projection, the method is built to be insensitive to the "modality gap," the geometric separation between image and text embeddings, which can distort methods that measure distance to a text anchor.
Core Claim
The claim has two testable parts. First, when patch features are reconstructed through the centered text subspace plus a very small learned residual, defects should produce larger reconstruction residuals than nuisance changes such as lighting, shadows or background texture. This separation should be wider than with an equal-rank bottleneck chosen by variance, either through principal component analysis or a learned low-rank projection. Second, detectors that score patches by their distance to a normality text prompt should change measurably when the modality gap is artificially widened or narrowed. Projection onto a centered text subspace should be invariant to that shift by construction.
If the first part fails, the text adds nothing beyond rank. That failure is plausible, because patch-level image-text alignment in contrastive models is weak, and the experiments are designed so this outcome shows up clearly.
Grounding in Prior Work
Two lines of evidence motivate the proposal. In pixel-based control, Tomar et al. show that full-image reconstruction objectives such as Dreamer spend their capacity on background distractors and collapse in uncurated scenes. McInroe et al. show that this distractor sensitivity hides small task-critical details. CoRAL responds by reserving reconstruction for a clean modality (proprioception) and using contrastive losses for noisy pixels. We borrow the principle that the cleaner modality should decide what gets reconstructed. In our setting, however, text defines the subspace, while the reconstruction still happens in visual features, because unsupervised detection has no reward or proprioceptive signal to fall back on.
In anomaly detection, ShortcutBreaker proves that a projection whose rank is below the feature dimension cannot learn the identity map. It combines a low-rank noisy bottleneck with perturbed attention and reports 99.8% image-level AUROC on MVTec AD. PIRN reconstructs through K=10 learned prototypes per modality and notes that this fixed K degrades on structurally complex objects. Both supply the mechanism we reuse, a rank-limited reconstruction path. Both choose that path by a scalar hyperparameter and by data variance, which is the part we replace.
One paper constrains the design. Do Autoencoders Need a Bottleneck for Anomaly Detection? finds that overcomplete Bayesian autoencoders can beat bottlenecked ones. We take this seriously: the bottleneck may not need to be narrow, only aimed correctly. For that reason we sweep the size of the learned residual, including large values, instead of assuming that narrower is better.
The closest work combines text and anomaly detection. CNC uses learnable "normal [object]" prompts in CLIP's joint space as a normality anchor. Visual features are modulated by their similarity to that anchor, and a mixture of expert decoders handles multiple classes. CNC reports 98.6% image-level AUROC on MVTec AD and 93.2% on VisA. It is a distance-to-a-point method, and it does not measure the modality gap. CMG uses paired captions to train a module that masks redundant image regions. It works at the image level on caption datasets and needs captions for every training image. Our proposal differs from both in three ways: text defines a subspace rather than a target point, no per-image captions are required, and the gap is explicitly removed and tested.
The gap analysis rests on two papers. Mind the Gap shows that contrastive image and text embeddings occupy disjoint cones separated by a largely shared offset, and that shifting along this offset changes downstream behavior. ModGap shows that rank-based operations such as retrieval are unaffected by the gap, while distance-based operations such as clustering are hurt by it. Anomaly scoring by distance to a text anchor is distance-based, which is why we predict gap sensitivity for CNC-style scoring. Subtracting each modality's mean is the cheapest way to remove the shared offset, and it is what makes our projection gap-invariant.
Research Program
The detector uses a frozen CLIP image encoder. From it we take patch tokens at several depths, and the final-layer patch tokens are mapped into the joint space with CLIP's own projection. For each category we write prompt sets in fixed templates, of the form "a [object] with [normal attribute]" and "a [object] under [normal condition]". The attribute lists cover expected parts, materials, orientations and benign variations. We encode the prompts, subtract the mean text embedding, and take an orthonormal basis $T$ of their span. Its dimension $k$ is set by a singular-value energy threshold fixed once across all categories.
An encoder maps each patch, with its spatial context, to a code $h$. The bottleneck keeps
$$z = T T^\top (h - \mu_v) + R\,g(h),$$
where $\mu_v$ is the mean normal image embedding, which removes the visual side of the gap. $TT^\top$ projects onto the text span. $R$ is a learned matrix of small rank $r$, and $g$ is a small network, together giving a residual path for normal detail the text does not name.
A decoder reconstructs the multi-depth patch features from $z$. Training uses only normal images, with a cosine reconstruction loss. The anomaly map is the per-patch reconstruction error, upsampled to pixels, and the image score is its maximum. Setting $r=0$ gives the pure text bottleneck. Replacing $T$ with other $k$-dimensional bases gives the controls described below.
The analysis component measures where capacity goes. We build nuisance test sets by applying lighting, white-balance, blur and background-texture perturbations to held-out normal images. We then report nuisance residual energy, the mean anomaly score on these perturbed normal images, against defect residual energy on real anomalies. This ratio is the direct measure of whether the bottleneck spends capacity on meaning or on variance.
Baselines
The naive reference point is the same frozen CLIP features scored with no learned bottleneck. Concretely, this is patch-to-nearest-normal-patch distance, together with plain zero-shot CLIP prompt matching of "normal" against "defective" descriptions. Any learned bottleneck should clear this.
The strongest existing reconstruction method is ShortcutBreaker. It reports 99.8% image-level AUROC on MVTec AD, 90.6% AUROC on Real-IAD and 87.8% on its universal medical benchmark. For VisA the file gives no explicit number, so there is no reported result found on Lacuna for ShortcutBreaker on VisA. We will run it twice: once with its own released configuration and backbone, and once with its low-rank bottleneck placed on our frozen CLIP features at rank equal to $k+r$. In both runs we use the same training images, epochs and hyperparameter search budget as our method. The second run separates the effect of the backbone from the effect of the bottleneck.
The strongest text-guided competitor is CNC. It reports 98.6% image-level AUROC on MVTec AD and 93.2% on VisA, and we run it on the same backbone and budget. For pixel localization the reference is DRÆM, which reports 98.0 image AUROC and 68.4 pixel AP on MVTec AD against PaDiM's 95.5 and 55.0. PIRN is the comparison for the prototype bottleneck in the few-shot setting.
The ablations that isolate the key ingredient keep everything fixed except the basis $T$. In turn, $T$ is replaced by a random $k$-dimensional subspace, by the top-$k$ principal components of normal patch features, and by text embeddings of irrelevant prompts. A separate ablation omits mean-centering.
Because MVTec AD is near saturation, beating ShortcutBreaker's image-level number there is not the target. The proposal counts as successful under three conditions. Text-spanned bottlenecks must beat equal-rank principal-component and low-rank-learned bottlenecks on VisA and Real-IAD pixel AU-PR, with gains consistent across seeds. They must show a lower nuisance-to-defect residual ratio. And they must stay within noise of ShortcutBreaker and CNC on image-level AUROC.
Why not just…?
Why not just use a memory bank of pretrained patch features? Nearest-neighbor scoring against stored normal patches is simple and strong. PIRN reports that memory banks degrade in few-shot regimes, where unseen normal variation gets flagged, and DRÆM reports PaDiM below reconstruction on MVTec AD pixel AP. A memory bank also has no mechanism for ignoring nuisance shifts, since a relit normal part is far from every stored patch. Our nuisance test sets measure exactly this. The memory-bank baseline runs on identical features at full-shot and 5-shot, and if it matches us on nuisance robustness, the claim fails.
Why not just tune the rank of a unimodal low-rank bottleneck? This is the most serious threat. The real question is whether the text-derived directions beat variance-derived directions at the same rank, or whether rank alone does the work. The equal-rank principal-component and ShortcutBreaker-on-CLIP controls answer it directly. If they match the text-spanned bottleneck on pixel AU-PR and on the nuisance ratio, we will conclude that language adds nothing here and report that result.
Why not just close the gap with ModGap-style fine-tuning and keep CNC's anchor? That would fix the distance distortion, but it changes the backbone and still scores by distance to a single point. A single point cannot represent the many ways a normal object varies. Mean-centering achieves gap removal without any training. We will include a CNC variant with centered embeddings, so if centering alone explains the gains, the subspace part of the proposal is shown to be unnecessary.
Why not just use zero-shot CLIP prompts? Global prompt matching faces the granularity mismatch described by the visual attention grounding work: small defects are overwhelmed by global context. The naive baseline includes this comparison.
Decisive Checks
We train on MVTec AD, VisA, Real-IAD and MVTec 3D-AD (RGB only, 5-shot). We report image-level AUROC, pixel AUROC and pixel AU-PR, along with the nuisance-to-defect residual ratio.
Three results decide the proposal. The first is the matched-rank comparison of text basis against principal-component, random and irrelevant-prompt bases on VisA pixel AU-PR. If the text basis does not win, the central claim is false. The second is a gap sweep. Following Mind the Gap, we shift text embeddings along the gap vector by a factor in a range around the natural gap, and we plot AUROC for CNC-style anchoring and for our projection. We predict that anchoring curves vary while ours stays flat. The third is a sweep of residual rank $r$ from zero to large. It tests, in the spirit of the no-bottleneck result, whether the text span still helps once the residual path is wide enough to permit near-identity reconstruction. If it does, the method resolves the tension between rank-limited and overcomplete designs: the bottleneck's direction matters more than its width.
References
- Learning Representations for Pixel-based Control: What Matters and Why?
- Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning
- Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL
- ShortcutBreaker: Low-Rank Noisy Bottleneck with Global Perturbation Attention for Multi-Class Unsupervised Anomaly Detection
- PIRN: Prototypical-based Intra-modal Reconstruction with Normality Communication for Multi-modal Anomaly Detection.
- Do Autoencoders Need a Bottleneck for Anomaly Detection?
- CNC: Cross-modal Normality Constraint for Unsupervised Multi-class Anomaly Detection
- Improving Vision Anomaly Detection With the Guidance of Language Modality
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Closing the Modality Gap Aligns Group-Wise Semantics
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- A Visual Attention Grounding Neural Model for Multimodal Machine Translation