Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs

Zhikang Xu, Qianqian Xu, Zitai Wang, Cong Hua, Sicong Li, Zhiyong Yang, Qingming Huang

TL;DR

InterNeg is a new framework that significantly improves out-of-distribution (OOD) detection for Vision-Language Models (VLMs) like CLIP. By ensuring that negative samples are selected based on inter-modal distances (image-to-text) rather than just text-to-text distances, it achieves state-of-the-art performance, including a 3.47% reduction in false positives on ImageNet and a 5.50% boost in accuracy on challenging "Near-OOD" tasks.

What Problem Does This Paper Address?

In the "open-world" deployment of AI, models often encounter objects they’ve never seen before. Ideally, a model should flag these as out-of-distribution (OOD) rather than confidently misclassifying them into a known category.

Recent research has turned to Vision-Language Models (VLMs) like CLIP for this task. Because CLIP is pre-trained on massive amounts of internet data, it already "knows" about millions of concepts. Current methods (like NegLabel) try to detect OOD samples by comparing a test image against a set of "negative" text labels (e.g., if an image looks more like "ash" or "dill" than the known class "cat," it’s probably OOD).

However, the authors identify a critical flaw: Distance Inconsistency. Most existing methods select these negative texts based on intra-modal distance—how far the text "ash" is from the text "cat" in the language space. But CLIP is optimized for inter-modal alignment—how well an image matches a text. Because the mapping between image and text spaces isn't perfectly identical, a text that seems "far" from a label might actually be "close" to an image of that label, leading the model to mistakenly flag known objects as OOD.

Methods

The authors propose InterNeg, which enforces consistency by using inter-modal distances throughout the detection process. The framework consists of two main strategies:

  1. Inter-modal Guided Negative Text Selection: Instead of picking negative texts that are semantically distant from ID (In-Distribution) labels, InterNeg picks texts that are distant from ID image proxies (the average visual features of a few example images). This ensures the "negatives" are truly visually distinct from the known classes.
  2. Inter-modal Guided Extra Negative Texts: During inference, if the model encounters an image it is highly confident is OOD, it performs modality inversion. It optimizes a set of "pseudo-tokens" to create a new text embedding that describes that specific OOD image. This dynamically expands the model's "negative vocabulary" on the fly.
  3. Dynamic Filtering: To prevent noisy or incorrect OOD images from ruining the results, the model uses a filtering mechanism. It only keeps the "inverted" text embeddings if they maintain a clear inter-modal distance from the known ID image proxies.

Datasets

The researchers evaluated InterNeg against a wide array of benchmarks:

  • In-Distribution (ID): ImageNet-1K, CIFAR-10, and CIFAR-100.
  • OOD Benchmarks: The "Traditional Four" (iNaturalist, SUN, Places, and Textures).
  • OpenOOD Benchmark: A comprehensive suite that distinguishes between "Near-OOD" (objects very similar to ID classes) and "Far-OOD" (highly distinct objects).

Key Insights and Ideas

  • The "Modality Gap" Matters: The most important insight is that the text-to-text distance is a poor proxy for image-to-text distance in CLIP. Aligning the detection strategy with CLIP’s training objective (inter-modal alignment) is the key to unlocking better performance.
  • Two Types of Errors: The authors categorize ID misclassification into "Max-OOD dominant" (one very bad negative text causes a mistake) and "Sum-OOD dominant" (many mediocre negative texts aggregate to cause a mistake). InterNeg's selection criteria effectively mitigate both.
  • Data Efficiency: Remarkably, the "image proxies" used for selection don't require a full training set. The authors found that using as few as 4 to 16 images per class is enough to create effective proxies, making this highly practical for low-data scenarios.

Key Results

InterNeg sets a new standard for zero-shot OOD detection (methods that don't require re-training the model):

  • ImageNet-1K: Achieved a 3.47% reduction in FPR95 (the rate of false positives when 95% of true positives are detected) compared to previous leaders.
  • Near-OOD: On the challenging Near-OOD benchmark, it showed a 5.50% improvement in AUROC, proving it can distinguish between very similar categories (like different breeds of dogs) better than previous methods.
  • Competitive with Supervised Methods: Perhaps most impressively, InterNeg often outperforms methods that do require training on ID data, despite being essentially a zero-shot approach.

Limitations and Future Work

  • Inference Latency: The "modality inversion" step (turning an OOD image into a text embedding) requires an optimization loop during inference. While the authors describe the overhead as "minor" (roughly 0.0067 seconds per image on an RTX 3090), it is still slower than a simple forward pass.
  • Proxy Dependency: While it only needs a few images, it still requires some visual examples of the ID classes to build the proxies. A purely text-based version that maintains this inter-modal consistency remains an open challenge.
  • Future Directions: The authors suggest exploring more efficient ways to perform modality inversion and investigating if this "inter-modal consistency" principle can improve other VLM tasks like zero-shot retrieval or grounding.

Summary updated

Abstract

Out-of-distribution (OOD) detection seeks to identify samples from unknown classes, a critical capability for deploying machine learning models in open-world scenarios. Recent research has demonstrated that Vision-Language Models (VLMs) can effectively leverage their multi-modal representations for OOD detection. However, current methods often incorporate intra-modal distance during OOD detection, such as comparing negative texts with ID labels or comparing test images with image proxies. This design paradigm creates an inherent inconsistency against the inter-modal distance that CLIP-like VLMs are optimized for, potentially leading to suboptimal performance. To address this limitation, we propose InterNeg, a simple yet effective framework that systematically utilizes consistent inter-modal distance enhancement from textual and visual perspectives. From the textual perspective, we devise an inter-modal criterion for selecting negative texts. From the visual perspective, we dynamically identify high-confidence OOD images and invert them into the textual space, generating extra negative text embeddings guided by inter-modal distance. Extensive experiments across multiple benchmarks demonstrate the superiority of our approach. Notably, our InterNeg achieves state-of-the-art performance compared to existing works, with a 3.47% reduction in FPR95 on the large-scale ImageNet benchmark and a 5.50% improvement in AUROC on the challenging Near-OOD benchmark.
BibTeX
@misc{xu2026mind,
  author = {Zhikang Xu and Qianqian Xu and Zitai Wang and Cong Hua and Sicong Li and Zhiyong Yang and Qingming Huang},
  title = {{Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMs}},
  year = {2026},
  eprint = {2603.02618},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02618},
}

Loading related content...

Figure