How Much Must an Image Change to Become a Known Class? A Style-Robust Near-OOD Score

Image classifiers are routinely shown images of things they were never trained on. When the unfamiliar image is very different, such as a truck shown to a dog-breed classifier, today's detectors flag it reliably. When it is close, such as a wolf shown to a husky classifier, they mostly fail. This harder case is called near-out-of-distribution (near-OOD) detection. Across more than forty methods on ImageNet, detectors reach about 95% AUROC (area under the ROC curve, where 50% is chance) on distant outliers but stall near 78% on near-OOD sets (OpenOOD v1.5). Two things make the gap stubborn. First, the evidence that separates a wolf from a husky lives in a few image regions, such as the muzzle, ears, and coat, and a single global embedding averages it away. Second, detectors confuse changes in style with changes in content. A known class rendered in a new texture or lighting gets rejected, while a novel class photographed in the familiar style slips through (Rethinking the Evaluation of OOD Detection: A Sorites Paradox).

We propose to score an image by asking a generative question rather than measuring a distance. The question is: what is the smallest edit that turns this image into a convincing member of some known class, and how much of the object does that edit touch? A test image from a known class needs almost no edit toward its own class. A near-OOD image needs a small but real edit to its decisive parts. A known class in an unusual style should also need almost no edit, provided the editor was trained to keep everything that is not class-relevant. That provision is where the other two directions enter. The editor is an unpaired image-to-image translator (a generator that learns to map images between categories without pixel-aligned before/after pairs). It is trained only on known classes, and a round-trip "cycle" constraint teaches it to leave non-semantic content alone. Bidirectional cross-modal attention, in which image patches and class-name text tokens attend to each other, decides which regions count as "the object" when we measure how much changed.

Core Claim

The claim is that the size of a minimal, class-conditioned, style-preserving edit, measured only inside the regions that the class text claims, separates near-OOD images from known-class images better than confidence or embedding-distance scores do. It should also degrade much less under pure style shift. The editor is used only as a test-time probe and is never used to produce synthetic outliers for training. This lets the method step around a live disagreement in the literature. MM-OOD argues that generative models should not be used for near-OOD because generated samples fall too close to the known classes. SynOOD and GCOS argue that synthesizing near-boundary outliers is exactly what helps. We never train the detector on generated images, so the contamination worry does not apply. Closeness to the known classes is the very quantity we measure.

We do not claim this escapes the impossibility result of Is Out-of-Distribution Detection Learnable?. That result says detection fails when known and novel data overlap in the chosen feature space. Our bet is narrower. We expect a class-conditioned edit residual, localized in space, to be a feature in which near-OOD classes overlap less than they do in pooled CLIP embeddings. Galil et al.'s severity-graded benchmark lets us test this directly (A framework for benchmarking Class-OOD detection).

Research Program

The detector has three stages.

Stage 1: candidate classes. A frozen CLIP model proposes the top-$k$ known classes for the input $x$, with $k$ around 5.

Stage 2: localization. For each candidate class $c$, we run CALIP-style parameter-free bidirectional attention (CALIP). CALIP lets CLIP's spatial grid tokens and text tokens attend to each other with no learned projections. It was evaluated only on closed-set zero-shot classification. We use its two directions for a different purpose:

  • In the text-to-image direction, the class text attends to the grid, which yields the regions the class claims.
  • In the image-to-text direction, each patch attends over all candidate class texts, which yields how exclusively that patch is explained by $c$.

Their product gives a soft mask $M_c$ over the image. A one-directional mask, built from the first direction alone, is kept as an ablation. The bidirectional version should suppress background that is shared across candidate classes, which is the "text-aware visuals" effect CALIP reports.

Stage 3: the class editor. Each candidate class gets an edit $E_c(x)$ from a text-conditioned editor. The editor starts from UIP2P, which trains instruction-based editing without paired data through an edit-reversibility cycle: a forward edit followed by a reverse edit must reconstruct the input. We fine-tune it on known-class images only, as unpaired translation between known classes ("make this a Siberian husky" applied to a malamute image, cycled back). We add an identity term that asks $E_c(x)\approx x$ whenever $x$ already belongs to class $c$. This extends the "golden template" identity loss of Kim et al.'s CycleGAN defect segmentation from periodic textures to semantic classes. Kim et al. detect defects by subtracting a translated defect-free template. We carry that subtraction idea into multi-class natural images and add language conditioning, which their unimodal setup lacks. We also augment the identity term with style-transferred known-class images, so that the editor learns to treat style as something it must not change.

The score. With $\phi$ denoting CLIP patch features and $\mu_c$ a per-class normalizer estimated on held-out known-class images, the score is

$$ s(x)=\min_{c\in \text{top-}k}\ \frac{1}{\mu_c}\,\frac{\sum_{p} M_c(p)\,\lVert \phi_p(x)-\phi_p(E_c(x))\rVert}{\sum_p M_c(p)}. $$

Here $p$ indexes grid positions. The numerator is the feature change inside the mask, normalized by the mask's total weight. High $s(x)$ means that even the easiest known class requires substantial changes to the regions that define it. The per-class normalizer $\mu_c$ absorbs the fact that some classes are simply more variable and harder to edit.

Cost control. Diffusion editing is expensive. We therefore run the detector as a cascade. A cheap first-stage score, SimLabel or MCM, settles confident cases. Only images in its uncertain band are sent to the editor. We report latency and the fraction of images escalated, because inference cost is a known obstacle for heavier detectors.

Baselines

Naive reference. The do-nothing point is maximum-softmax / MCM scoring on frozen CLIP. This is the reference against which SimLabel-I reports an ImageNet-1K FPR95 of 43.61%.

Strongest post-hoc competitor. The strongest training-free competitor on the same backbone is SimLabel. It scores an image by whether its similarity is coherent across a neighborhood of related class names. SimLabel-I reports 91.90% AUROC and 36.46% FPR95 on ImageNet-1K, and 2.25% FPR on CUB-200. It is the right competitor because it uses the same frozen CLIP, needs no training, and targets exactly the problem that near-OOD images match one label without matching its semantic neighbors. The ImageNet figures are averaged over mostly distant outlier sets. We found no reported result on Lacuna for SimLabel on SSB-hard or NINCO. We will therefore run it ourselves on those sets with the same CLIP ViT-B/16 and the same prompt templates. Beating it means higher AUROC on SSB-hard and NINCO and lower FPR95 on CUB-200 near-OOD splits, averaged over three seeds of any stochastic component.

Strongest trained, generative competitor. SynOOD trains a light head on diffusion-synthesized near-boundary images. It reports a 2.80% AUROC gain and an 11.13% FPR95 reduction on ImageNet over its base. Absolute near-OOD numbers are not given in the file, so we rerun it. For fairness, SynOOD's generation budget is matched to our editor fine-tuning budget in GPU-hours, and both use the same known-class training images.

Strongest feature-geometry competitor. HVCM models each class as Gaussian sub-concepts and scores by aggregated Mahalanobis distance. It reports 21.99% FPR95 and 92.73% AUROC on ImageNet-1K.

Saturated benchmark. GCOS reports 0.00% FPR95 on Stanford Dogs. That benchmark is therefore saturated and cannot show a gain, so we use it only as a sanity check.

Ablations isolating the key ingredients. Four ablations each remove one ingredient:

  • Replacing $M_c$ with a uniform mask tests whether localization matters.
  • Replacing bidirectional attention with text-to-image attention only tests whether the second direction matters.
  • Using the off-the-shelf UIP2P editor without known-class fine-tuning and identity loss tests whether the cycle-plus-identity training matters.
  • Dropping the editor entirely and scoring with the CALIP masked patch-text similarity alone tests whether generation matters at all.

Why not just…?

Why not just use SimLabel or MCM on the global CLIP embedding? These scores are cheap and strong. However, they compare one pooled vector to text, which is precisely where localized near-OOD evidence is averaged away. They also inherit CLIP's confusion of style with content, which the Sorites paper documents for global-embedding detectors. The Syn-IS style grid will show whether that inherited confusion costs them near-OOD accuracy relative to our score. If SimLabel matches us on SSB-hard and NINCO and on the style grid, the editor is not earning its cost and we will say so.

Why not just apply CALIP's bidirectional attention to OOD scoring, with no generator? This is the most serious threat. The localization alone might carry most of the gain, and it is nearly free. We run it as the "no-editor" ablation under the same masks. The generator is justified only if it adds a clear margin on the near-OOD sets and on style-shifted known classes. The reasoning for why it might is that attention similarity says only "this patch resembles husky." The edit residual says "this patch would have to change to be husky," and that is a comparison against the known-class manifold rather than against a single text vector. If the margin is small, the right outcome is a paper about attention-localized scoring, and we will write that one.

Why not just use reconstruction error from an unconditional autoencoder or diffusion model? Reconstruction methods are known to detect gross anomalies but to lack the precision needed to separate close classes. In Kim et al., translation-based residuals collapse to 12–14% mIoU once the data stop being rigidly periodic. Unconditional reconstruction also has no notion of which content is class-defining, so style shifts inflate the error. We include this baseline to test that expectation. Our score's defense is class conditioning and masking, not generation itself.

Why not just train with synthetic outliers, as GCOS or SynOOD do? These methods reshape the decision boundary using known classes plus synthesized negatives. They remain tied to a single feature space and to style-sensitive training images. Our score is complementary and could sit on top of either. We report SynOOD alone, our score alone, and the two combined, to show whether the probe adds information that training on synthesized negatives does not.

Proposed Experiments and Decisive Checks

Main evaluation. The main table uses the OpenOOD v1.5 ImageNet-1K near-OOD sets, SSB-hard and NINCO, where the reported plateau is about 78% AUROC. It adds CUB-200 and ImageNet-10/20. MM-OOD reports 3.84% FPR95 on ImageNet-10; that figure uses an MLLM, so compute is reported alongside it. Metrics are AUROC, FPR95, and in-distribution accuracy.

Graded difficulty. We plot AUROC against severity level $s_0$ through $s_{10}$ on Galil et al.'s graded benchmark. The hypothesis predicts that our advantage grows with severity, since the hardest levels are where outliers land inside known-class clusters in embedding space. A flat or shrinking advantage refutes it.

Style robustness. This is the decisive check for the style-preservation claim. We use IS-OOD and Syn-IS, which contain 320,000 SDXL-Turbo images spanning controlled styles. For each style, we measure the false-rejection rate of known-class images at the operating threshold that yields 95% true positives on standard-style known-class data. The claim requires two things. Our score's false-rejection rate should rise far less across styles than MCM's and SimLabel's. Its near-OOD detection should stay flat when the novel classes are rendered in the training style. Removing the style-augmented identity term should reintroduce the gap; if it does not, the style robustness is not coming from the cycle training.

Cost. We report per-image latency, the fraction of images escalated by the cascade, and a curve of accuracy against escalation fraction. This shows whether a modest escalation rate, for example the least certain 20% of images, captures most of the gain.