Industrial anomaly detectors learn best when they see examples of what "broken" looks like, but real defects are rare. Current practice makes fake ones. Some methods paste and blend image patches; others run generative image models, which are far slower. This proposal tests a much cheaper source. In a vision-language model such as CLIP, a model trained so that images and their captions land near each other in one shared vector space, the text "a photo of a scratched metal nut" minus the text "a photo of a normal metal nut" is a single vector. It costs two text-encoder calls. If that vector points in roughly the same direction as the change a real scratch causes in the image features, we can add it to features of normal image patches and get synthetic anomalies with no pixels rendered at all.
The obstacle is a well-documented property of these models called the modality gap: image vectors and text vectors sit in separate regions of the shared space, so text vectors cannot simply be dropped into image space. Our central observation is that subtracting a "normal" reference inside each modality should cancel most of this gap. The gap is largely a constant offset, and a constant offset disappears when you take a difference. So the core claim is testable and falsifiable. After residualization, meaning after subtracting the normal reference, text-derived defect directions become usable anomaly generators in image-feature space. They should beat norm-matched random perturbations for detecting defects on unseen object classes, at a synthesis cost several orders of magnitude below pixel-space pipelines.
Background and What Is Missing
Three lines of work meet here, and each plays a different role.
Residual-based anomaly detection supplies the method. ResAD subtracts from each test patch feature its nearest neighbor among a few normal reference patches. It then scores these residuals with a shallow network trained under a one-class loss plus a normalizing flow, reaching 83.0% average image-level AUROC in a 4-shot, class-generalizing setup, and 88.0% when the backbone is ImageBind. NAGL mines residuals between normal and a few abnormal references to form "residual proxies." Both are purely visual; NAGL's authors explicitly list language prompts as future work. CRAS already synthesizes anomalies in feature space rather than pixels, perturbing features with variance scaled by their distance to class centers. Its perturbation directions carry no semantics, however, and it never touches text. CRAS is the closest prior art: our proposal keeps its "synthesize in latent space" design and asks whether language can tell us which directions to perturb in.
Modality-gap research supplies the geometry and motivates the cancellation argument. Connect, Collapse, Corrupt models paired image and text embeddings as differing by a constant vector plus Gaussian noise. It shows that subtracting modality means and adding noise lets decoders trained on one modality work on the other. Mind the Gap traces the gap to narrow-cone geometry and contrastive training, and shows that tuning its size can help downstream tasks. It's Not a Modality Gap attributes it to low uniformity of the embedding distribution and closes the centroid distance from 0.66 to 0.08 on MS-COCO with extra losses. None of these papers study differences of embeddings, and none study anomaly detection. There is also a warning from a related direction: ViM finds that feature components outside the classifier's main subspace carry out-of-distribution signal. Aggressively projecting away "gap" directions might therefore erase exactly the signal we want. The proposal must measure this rather than assume it away.
Synthetic-data overhead supplies the constraint and one axis of evaluation. CLIP-FSAC generates pixel-level anomalies by random perturbation and Poisson-blended patches, then trains cross-modal adapters in two alternating stages. DISCO shows in NLP that filtered synthetic examples beat much larger unfiltered sets, but each candidate costs an extra teacher-model pass. Our goal is a synthesis step whose cost is dominated by a few hundred text-encoder forward passes, with no rendering and no filtering model.
Core Claim
Let $v(\cdot)$ and $t(\cdot)$ be the image-patch and text encoders of a shared-space model. Suppose, following the Connect, Collapse, Corrupt model, that semantically matched image and text inputs satisfy $v(x) - t(y) = c + \epsilon$, where $c$ is a constant gap vector and $\epsilon$ is noise. Then, for an anomalous patch $x_a$ with normal reference $x_n$, and a defect prompt $y_a$ with normal prompt $y_n$,
$$\big(v(x_a) - v(x_n)\big) - \big(t(y_a) - t(y_n)\big) = \epsilon_a - \epsilon_n ,$$
so the constant gap cancels exactly and only noise remains. The prediction is that visual residuals $\Delta_v = v(x_a) - v(x_n)$ and text residuals $\Delta_t = t(y_a) - t(y_n)$ should share direction up to noise and a scale factor. Raw text embeddings should not. We use this to synthesize anomalous features
$$\tilde z = z + \alpha\, s\, \hat\Delta_t ,$$
where $z$ is a normal patch feature, $\hat\Delta_t$ is a unit-norm text residual sampled from a bank of defect prompts, $s$ is a scale matched to the spread of normal-patch residuals computed from the few normal shots, and $\alpha$ is a severity factor drawn from a fixed range. The synthetic residuals then train a ResAD-style shallow scorer as positive examples of abnormality.
The claim fails if one of two things happens. Real defect residuals may show no more alignment with text residuals than with random directions. Alternatively, text-residual synthesis may not beat norm-matched isotropic noise injected in the same place. We treat either outcome as a publishable negative result.
Research Program
The first stage is a geometric study that requires no new training. Using CLIP ViT-L/14 with WinCLIP-style window features, and ImageBind (the backbone ResAD found strongest), we compute visual residuals for the real annotated defects of MVTec AD and VisA. This analysis uses held-out ground truth only; it is never used for training. For each defect type we build a prompt bank: state words such as "scratched," "cracked," "missing part," or "contaminated," crossed with the templates of WinCLIP. From these we form text residuals.
We then measure four things:
- the cosine between each real visual residual and its matched text residual, against three controls: mismatched defect prompts, random-word prompts, and random directions;
- the principal angles between the subspace spanned by real defect residuals and the subspace spanned by text residuals;
- whether residualization removes the mean offset, by comparing mean($\Delta_v$) − mean($\Delta_t$) with the raw embedding gap;
- how much defect energy lies outside the top principal directions of normal features, which is the question ViM raises.
The second stage builds the synthesizer and detector described above, with three design choices studied explicitly. The first is how much of the residual mean offset to keep. We sweep from full removal to none, since Mind the Gap suggests the best gap need not be zero. The second is whether to project $\hat\Delta_t$ onto, or away from, the normal principal subspace. The third is whether a light per-class calibration, a single scalar and sign fitted by maximizing separation from the normal-residual distribution, adds anything over pure norm matching. No stage uses real anomalous images for training.
The third stage accounts for cost. We report GPU-seconds and FLOPs per thousand synthetic anomalies, end-to-end training wall-clock time, and inference frames per second, all on the same hardware for every synthesis method.
Baselines
The naive reference point is ResAD with the same backbone and the same few normal shots but no synthetic anomalies. Its reported figures are 83.0% average image-level AUROC at 4 shots with WideResNet50 and 88.0% with ImageBind (ResAD). We rerun it under our exact class-generalizing protocol: train on MVTec AD, test on VisA, BTAD, MVTec3D and BraTS, and the reverse direction. Any gain we claim must be relative to this rerun.
The key ablation, and the most dangerous competitor, is isotropic Gaussian feature perturbation injected at the same location. Its per-sample norm is matched to our $\alpha s$ distribution, and it uses the same scorer, epochs and severity sweep. We also include a CRAS-style distance-guided perturbation, reimplemented on our backbone following CRAS. CRAS reports 98.3% image-level and 98.0% pixel-level AUROC on MVTec AD, but in a multi-class setting trained on all classes, not a class-generalizing one, so its number is not directly comparable. Two further semantic controls isolate the ingredient itself: residuals from random-word prompts, and raw defect-text embeddings without subtracting the normal prompt. Beating the norm-matched isotropic control is the result that matters. If text residuals win only by the amount random-word residuals also win, the gain comes from anisotropy, not language.
The strongest existing system is CLIP-FSAC, which uses pixel-level synthetic anomalies with cross-modal adapters. It reports 96.0% image-level AUROC on VisA in the 1-shot setting and 95.5% on MVTec AD, the latter with the shot setting unspecified in the file (CLIP-FSAC). We run its synthesis step, random perturbation plus Poisson-blended patches, through our scorer, and also run the full system with the same backbone and the same normal shots. We tune each method with the same number of hyperparameter trials. Other reported reference points are WinCLIP+ at 1-shot (93.1% on MVTec AD, 83.8% on VisA) (WinCLIP), NAGL at 95.8% on MVTec AD with one normal and one abnormal reference, and CNC at 98.6% on MVTec AD and 93.2% on VisA in its multi-class setting (CNC).
Our success criterion has two parts. We must match CLIP-FSAC's pixel-synthesis accuracy within about one AUROC point on VisA and MVTec AD while cutting synthesis cost by at least two orders of magnitude. We must also beat the norm-matched isotropic control on unseen-class transfer by a margin larger than seed variance across five seeds.
Why not just…?
Why not just perturb features with Gaussian noise, as CRAS-style latent synthesis already does? This is the real threat. Latent noise is already nearly free, so text residuals cannot win on cost against it, only on accuracy. There are reasons to expect a gap. Real defects are structured, and CRAS itself reports that unstructured perturbations blur the normal boundary and cause false positives. But the file contains no direct evidence that semantic directions beat random ones in anomaly feature space. The matched isotropic ablation, together with the random-word-prompt control, settles this. If they tie, the conclusion is that language adds nothing here, and we report that.
Why not just synthesize defects in pixels, as CLIP-FSAC does with Poisson-blended patches or as diffusion-based methods do? Pixel synthesis is effective. CLIP-FSAC's 96.0% 1-shot VisA result is the bar we must meet. It pays per sample for rendering and a full image-encoder pass, however, and diffusion-based generation is much slower still (Direction 3; Beyond Cropping and Rotation). Blended patches also produce only the defect types that cut-and-paste can imitate. Our cost table and per-defect-type breakdown show whether the saving costs accuracy, and whether text reaches defect types, such as discoloration or missing parts, that patch pasting handles poorly.
Why not just engineer better prompts and score images zero-shot, as WinCLIP does? Prompt ensembles use the same text knowledge we do but only compare images to prompts at test time, where the modality gap works against them. WinCLIP's 1-shot VisA result of 83.8%, against CLIP-FSAC's 96.0%, suggests that training a scorer on synthetic anomalies matters. We include a zero-shot scorer that uses our residualized text directions directly, with no training, to separate "better use of text" from "training on synthesized examples."
Why not just close the modality gap with trained adapters or uniformity losses and then compare image and text directly? Adapters need stable training in few-shot regimes; CLIP-FSAC required an alternating two-stage schedule. Closing the gap through extra losses (It's Not a Modality Gap) retrains the encoders. Our cancellation argument says residualization removes the constant part of the gap with no training at all. If the constant-offset model is wrong for patch features, however, the adapter route may be necessary. The offset sweep and the geometric study would expose exactly that, and we would then add the gap-closing loss as a variant.
Decisive Checks
We treat three results as decisive:
- In the geometric study on MVTec AD and VisA, matched text residuals must align with real defect residuals significantly better than mismatched and random-word residuals, and the mean offset must shrink substantially after residualization. If neither holds, the mechanism is wrong regardless of downstream accuracy.
- Under the class-generalizing protocol, text-residual synthesis must beat the norm-matched isotropic and random-word controls on image-level AUROC, pixel-level AUROC and AU-PR on held-out datasets, including the industrial-to-medical transfer to BraTS where NAGL reported a 10.6-point gain over ResAD.
- The cost comparison must show whether matching pixel-level synthesis within a point is achievable at near-zero synthesis overhead.
The offset and subspace-projection sweeps additionally answer an open question on their own: how much modality gap is best to keep when the goal is isolating defect signal rather than classification.
References
- ResAD: A Simple Framework for Class Generalizable Anomaly Detection
- Normal-Abnormal Guided Generalist Anomaly Detection
- Center-Aware Residual Anomaly Synthesis for Multiclass Industrial Anomaly Detection
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
- ViM: Out-Of-Distribution with Virtual-logit Matching
- CLIP-FSAC: Boosting CLIP for Few-Shot Anomaly Classification with Synthetic Anomalies
- DISCO: Distilling Counterfactuals with Large Language Models
- WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation
- CNC: Cross-modal Normality Constraint for Unsupervised Multi-class Anomaly Detection
- Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Models